-- Logs begin at Sun 2022-08-21 00:10:36 UTC, end at Sun 2022-08-21 01:46:58 UTC. -- Aug 21 01:03:31.345848 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Aug 21 01:03:31.357659 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Aug 21 01:03:31.360745 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: *** Starting uWSGI 2.0.18-debian (64bit) on [Sun Aug 21 01:03:31 2022] *** Aug 21 01:03:31.360745 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Aug 21 01:03:31.360745 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Aug 21 01:03:31.360745 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: nodename: dv-cinder-master-3-iscsi Aug 21 01:03:31.360745 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: machine: x86_64 Aug 21 01:03:31.360745 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: clock source: unix Aug 21 01:03:31.360745 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: pcre jit disabled Aug 21 01:03:31.360745 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: detected number of CPU cores: 4 Aug 21 01:03:31.370107 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: current working directory: / Aug 21 01:03:31.370107 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: detected binary path: /usr/bin/uwsgi-core Aug 21 01:03:31.370107 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: your processes number limit is 47680 Aug 21 01:03:31.370107 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: your memory page size is 4096 bytes Aug 21 01:03:31.370107 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: detected max file descriptor number: 2048 Aug 21 01:03:31.370107 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: lock engine: pthread robust mutexes Aug 21 01:03:31.370107 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: thunder lock: enabled Aug 21 01:03:31.370107 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Aug 21 01:03:31.370107 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Aug 21 01:03:31.405141 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: Python main interpreter initialized at 0x5561fada09c0 Aug 21 01:03:31.405141 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: python threads support enabled Aug 21 01:03:31.405141 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: your server socket listen backlog is limited to 100 connections Aug 21 01:03:31.405141 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: your mercy for graceful operations on workers is 90 seconds Aug 21 01:03:31.405553 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: mapped 403077 bytes (393 KB) for 2 cores Aug 21 01:03:31.405553 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: *** Operational MODE: preforking *** Aug 21 01:03:31.405849 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: *** uWSGI is running in multiple interpreter mode *** Aug 21 01:03:31.405849 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: spawned uWSGI master process (pid: 81793) Aug 21 01:03:31.406071 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Aug 21 01:03:31.406952 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: spawned uWSGI worker 1 (pid: 81804, cores: 1) Aug 21 01:03:31.407052 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: spawned uWSGI worker 2 (pid: 81805, cores: 1) Aug 21 01:03:31.407567 dv-cinder-master-3-iscsi devstack@keystone.service[81793]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Aug 21 01:03:35.784027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 21 01:03:35.784027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: from cryptography.utils import int_from_bytes Aug 21 01:03:35.785041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 21 01:03:35.785041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: from cryptography.utils import int_from_bytes Aug 21 01:03:35.789539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 21 01:03:35.789539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: from cryptography.utils import int_from_bytes Aug 21 01:03:35.790098 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 21 01:03:35.790098 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: from cryptography.utils import int_from_bytes Aug 21 01:03:35.835419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 21 01:03:35.835644 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 21 01:03:35.835784 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] command line args: [] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 21 01:03:35.836055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 21 01:03:35.836258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ================================================================================ {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 21 01:03:35.836511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] admin_token = **** {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.836755 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] config_dir = [] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.836943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.837122 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] config_source = [] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.837304 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] debug = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.837504 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.837870 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.837957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 21 01:03:35.838066 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.838141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 21 01:03:35.838253 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] command line args: [] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 21 01:03:35.838335 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.838523 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 21 01:03:35.838601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.838705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ================================================================================ {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 21 01:03:35.838780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] list_limit = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.838918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] log_config_append = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.838993 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] admin_token = **** {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.839104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.839179 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] config_dir = [] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.839280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] log_dir = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.839425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.839533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] log_file = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.839597 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] config_source = [] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.839791 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.839895 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] debug = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.840572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.840664 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.840754 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.840958 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.841075 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.841154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.841231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.841231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.841372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.841450 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.841521 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.841704 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.841819 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] list_limit = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.841932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] log_config_append = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.842074 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.842186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.842292 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.842435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] log_dir = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.842540 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.842676 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] log_file = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.842800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.842912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.843054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.843160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.843272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.843407 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.843524 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.843664 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.843664 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.843664 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.843960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.47/identity {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.844132 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.844273 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] publish_errors = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.844423 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.844564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.844682 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.844818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.844931 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.845042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.845179 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.845290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.845405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.845649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.845726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.845830 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] standard_threads = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.845830 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.845977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.846098 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.846170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.846274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.846274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] use_journal = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.846274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] use_json = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.846487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.47/identity {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.846687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] use_stderr = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.846851 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] publish_errors = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.846959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] use_syslog = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.847037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.847104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.847173 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.847247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.847321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.847390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.847466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.847537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.847604 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.847678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.847746 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] standard_threads = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.847911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.847999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.848167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.848292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.848370 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.848439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.848507 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] use_journal = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.848656 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.848734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] use_json = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.848843 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] use_stderr = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.848921 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.849020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] use_syslog = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.849097 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.849252 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 21 01:03:35.849335 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.849510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.849597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.849754 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.849847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.850020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.850188 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.850332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.850470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.850607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.850769 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.850769 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.850981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.850981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.851174 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.851256 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.851367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.851480 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.851561 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.851634 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.851706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.851894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.851998 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.852151 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.856937 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.857047 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.857126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.857186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.857271 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.857271 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.857414 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.857491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.857562 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.857638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.857720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.857793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.857868 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.857939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.858014 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.858091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.858263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.858339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.858410 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.858479 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.858555 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.858627 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.858713 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.858804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.858880 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.859004 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.859124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.859205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.859287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.859360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.859436 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.859508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.859508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.859633 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.859711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.859783 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.859852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.859926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.859998 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.860073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.860196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.860272 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.860348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.860418 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.860491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.860552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.860620 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.860686 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.860745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.860807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.860896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.860967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.861028 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.861086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.861149 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.861207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.861264 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.861328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.861389 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.861447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.861504 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.861562 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.861631 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.861698 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.861765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.861835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.861916 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.861983 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.862056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.862127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.862203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.862279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.862358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.862425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.862495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.862563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.862642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.862720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.862792 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.862859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.862941 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.863015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.863083 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.863161 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.863230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.863315 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.863391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.863451 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.863520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.863589 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.863657 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.863787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.863865 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.863937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.864068 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.864288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.864371 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.864432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.864491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.864615 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.864744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.864823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.864942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.865019 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.865146 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.865231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.865359 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.865445 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.865567 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.865645 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.865723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.865841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.865917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.866067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.866144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.866263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.866341 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.866413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.866538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.866623 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.866698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.866823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.866901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.866973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.867051 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.867195 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.867300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.867365 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.867491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.867570 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.867643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.867771 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.867866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.868023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.868132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.868280 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.868366 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.868565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.868663 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.868796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.868884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.869025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.869106 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.869260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.869349 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.869489 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.869569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.869729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.869815 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.869965 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.870056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.870214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.870294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.870449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.870528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.870700 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.870787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.870940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.871019 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.871182 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.871273 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.871438 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] auth.external = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.871534 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.871668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.871748 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.871898 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.871976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.872149 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.872247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.872392 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.872478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] auth.password = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.872627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.872757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] auth.token = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.872870 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.872961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.873119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.873209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.873357 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.873489 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.873598 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.873693 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.873853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.873943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.874056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.874147 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.874286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.874367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.874523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] credential.caching = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.874620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] auth.external = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.874760 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.874839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.875009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.875093 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.875242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.875319 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.875600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] auth.password = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.875846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] auth.token = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.875927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.876010 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.876205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.876312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.876462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.876567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.876700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.876804 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.876953 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.877055 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.877238 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.877318 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.877483 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.877564 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.877745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.877838 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] credential.caching = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.877987 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.878073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.878223 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.878305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.878469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] federation.caching = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.878620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.878771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.878855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.878928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.879003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.879166 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.879269 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.879393 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.879464 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.879640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.879736 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.879877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.879956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.880147 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.880245 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.884866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.884960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.885041 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.885117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.885192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.885374 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.885471 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.885549 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity.caching = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.885623 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.885696 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.885768 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] federation.caching = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.885901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.885985 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.886059 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.886133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.886205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.886284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.886366 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.886442 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.886514 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.886586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.886658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.886729 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.886808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.886883 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.886883 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.887491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.887491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.887599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.887681 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.887756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.887828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.887900 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.887973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.888045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity.caching = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.888148 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.888225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.888298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.888370 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.888442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.888514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.888603 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.888680 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.888680 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.888812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.888885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.888965 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.889040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.889113 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.889113 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.889244 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.889317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.889397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.889477 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.889623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.889696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.889757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.889825 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.889899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.889960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.890021 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.890092 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.890164 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.890233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.890302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.890362 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.890433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.890504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.890577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.890657 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.890731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.890803 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.890876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.890953 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.891027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.891107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.891182 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.891262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.891342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.891416 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.891488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.891572 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.891648 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.891720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.891792 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.891863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.891935 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.892014 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.892089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.892232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.892310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.892394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.892471 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.892552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.892627 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.892699 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.892772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.892844 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.892916 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.892996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.893071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.893144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.893206 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.893272 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.893334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.893401 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.893463 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.893522 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.893660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.893735 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.893900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.893990 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.894140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.894306 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.894392 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.894543 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.894720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.894810 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.894984 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.895087 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.895216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.895294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.895446 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.895532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.895696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.895767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.895937 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.896007 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.896201 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.896295 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.896432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.896502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.896663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.896733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.896892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.896974 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.897136 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.897215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.897375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.897448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.897611 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.897690 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.897859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.897929 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.898090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.898161 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.898338 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.898417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.898576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.898648 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.898813 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.898896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.899047 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.899119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.899281 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.899351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.899527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.899658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.899865 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.899949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.900012 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.900235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.900310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.900490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.900575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.900735 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.900806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.900973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.901044 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.901209 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.901288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.901442 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.901511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.901686 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.901756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.901927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.902006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.902160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.902244 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.902395 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.902475 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.902692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] resource.caching = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.903440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.903561 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.903691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.903764 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.904011 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.904211 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.904301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.904366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.904483 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.904555 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.904722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.904793 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.904971 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.905056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.905208 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.905288 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.905477 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.905567 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.905684 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] role.caching = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.905872 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.906005 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.906132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] role.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.906132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.906378 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.906532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.906668 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.906804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.906942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.907072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.907201 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] resource.caching = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.907349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.907499 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.907635 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.907776 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.907911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.908033 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.908160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.908241 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.908315 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.908388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.908462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.908535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.908616 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.908725 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.908817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.908892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.908966 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.909115 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.909208 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.909353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] role.caching = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.909606 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] role.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.909687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.909839 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.909920 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.910092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.910173 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.910340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.910420 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.910587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.910675 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.910835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.910915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.911067 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.911147 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.911315 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.911402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.911562 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.911644 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.911799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.911878 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.912030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.912195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.912275 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.912348 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.912503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.912585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.912755 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.912886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.913016 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.913091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.913253 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.913425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.913506 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.913662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.913767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.913898 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.914034 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.914164 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.914302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] token.caching = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.914454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.914588 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.914726 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.914861 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.914998 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.915134 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.915134 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.915387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.915506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.915663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.915785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.915949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.916031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.916128 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.916206 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.916279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.916352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.916465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.916550 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.916724 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.916805 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.916958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.917037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.917199 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.917280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.917436 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.917517 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.917664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.917742 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.917899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] token.caching = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.917981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.918137 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.918218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.918423 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.918513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.connection = **** {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.918664 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.918759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.918923 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.919006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.919162 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.919241 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.919397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.919485 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.919632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.919712 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.919868 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.919950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.920155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.920255 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.920367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.920483 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.920605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.920687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.920838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.920927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.921081 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.921170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.921318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.921391 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.921544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.921624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.921786 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.921855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.922029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.922099 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.922258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.922353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.connection = **** {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.922499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.922662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.922740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.922814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.922887 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81804) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 21 01:03:35.923021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.923264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.923501 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.923737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.923987 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.924233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.924479 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.924715 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.925193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.925441 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.925674 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.925916 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.926147 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.926386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.926717 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 21 01:03:35.926910 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81805) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 21 01:03:35.953284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:35.953516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:35.953741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:35.962927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:35.964455 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:35.964688 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:35.964911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:35.974441 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:35.999955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.000207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.001242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.001468 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.058700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.059206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.115577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.115809 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.116232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.116473 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.272247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.272474 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.272700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.272916 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.273134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.273347 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.273477 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.273709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.273959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.274177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.274419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.274637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.308038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.308337 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.308469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.308579 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.308663 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.308808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.308946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.309031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.309107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.309192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.309269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.309343 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.309416 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.309610 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.320723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.321492 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.333315 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.333540 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.333756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.333970 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.334101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.334232 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.334361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.334512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.334649 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.334778 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81804) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.334905 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.334905 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.335145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.335328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81805) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 21 01:03:36.351826 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.352029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 21 01:03:36.352396 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.352596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 21 01:03:36.352707 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.352791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 21 01:03:36.352866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.353114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.353197 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.353327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 21 01:03:36.353605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.353687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.353884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.353967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.354301 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.354604 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.364143 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.364467 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.364709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.364984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.365237 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.365526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.365798 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.366083 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.371975 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.373096 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.374855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.375135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.375477 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.375693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.375980 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.376092 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.376265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.376390 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.376536 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.376677 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.376808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.376963 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.377182 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.377262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.377460 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.377543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.377727 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.377809 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.378026 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.378299 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.378575 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.378845 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.400191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.401663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.401946 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.402062 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.404130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.404416 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.407238 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.407526 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.407810 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.409205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.409493 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.409778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.414132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.414409 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.416391 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.416681 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.419062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.419350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.419631 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.419905 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.420197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.424355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.424355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.424355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.424610 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.424610 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.424610 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.424610 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.424610 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.424610 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.424991 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.424991 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.438106 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.438383 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.441149 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.442255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.445373 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.445704 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.445999 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.448616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.448919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.450307 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.453206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.453431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.456423 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.456655 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.456952 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.457234 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.457518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.457798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.458113 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.459787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.460092 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.460423 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.460712 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.460994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.461279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.474429 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.474719 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.474985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.475287 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.477824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.478141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.478434 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.478705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.482734 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.487454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.487454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.487624 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.487865 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.487945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.488041 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.488146 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.488340 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.490812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.491117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.491408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.491701 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.492060 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.499677 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.499955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.500260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.500542 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.500817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.501090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.501542 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.503713 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.504015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.504342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.504631 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.504916 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.505255 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.505552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.516788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.519794 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.521114 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.522730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.524243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.524338 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.525800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.526081 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.526412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.527166 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.529223 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.530775 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.531059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.531377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.533402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.534841 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.537786 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.538041 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.538231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.539005 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.539005 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.539547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.542526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.542822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.543163 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.543973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.547183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.547470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.547749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.552043 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.552359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.552644 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.554736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81804) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.555046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.555328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.555599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.555868 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.556159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.556446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.556712 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.556977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.557261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.557531 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.557790 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.558071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.558326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81804) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.559893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81805) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 21 01:03:36.560264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.560550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.560835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.561238 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.561674 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.562061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.562339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.562617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.562909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.563186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.563469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.563741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.564007 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81805) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 21 01:03:36.592982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5561fada09c0 pid: 81804 (default app) Aug 21 01:03:36.596612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ccab07-a2db-40a5-add1-76d3c7e41eff None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:36.596890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ccab07-a2db-40a5-add1-76d3c7e41eff None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:36.597156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ccab07-a2db-40a5-add1-76d3c7e41eff None None] PATH_INFO: `/v3` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:36.599582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5561fada09c0 pid: 81805 (default app) Aug 21 01:03:36.601350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1/1] 10.222.0.47 () {50 vars in 791 bytes} [Sun Aug 21 01:03:36 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:03:44.230230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8401c5c6-479d-4bf0-a613-b870c12f3492 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:44.230775 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8401c5c6-479d-4bf0-a613-b870c12f3492 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:44.230775 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8401c5c6-479d-4bf0-a613-b870c12f3492 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:44.235021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1/2] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:03:44 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:03:44.241845 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642ffdae-439f-4cf0-8b78-296a1f485bde None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:44.242299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642ffdae-439f-4cf0-8b78-296a1f485bde None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:44.242387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642ffdae-439f-4cf0-8b78-296a1f485bde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:44.312575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: INFO dbcounter [None req-642ffdae-439f-4cf0-8b78-296a1f485bde None None] Registered counter for database keystone Aug 21 01:03:44.334210 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG oslo_db.sqlalchemy.engines [None req-642ffdae-439f-4cf0-8b78-296a1f485bde None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81804) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 21 01:03:44.341318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writer thread running {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 21 01:03:44.524998 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG passlib.handlers.bcrypt [None req-642ffdae-439f-4cf0-8b78-296a1f485bde None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=81804) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 21 01:03:44.525343 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG passlib.handlers.bcrypt [None req-642ffdae-439f-4cf0-8b78-296a1f485bde None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81804) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 21 01:03:44.581263 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-642ffdae-439f-4cf0-8b78-296a1f485bde None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:03:44.582864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-642ffdae-439f-4cf0-8b78-296a1f485bde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:03:44.717802 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 2/3] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:03:44 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 477 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Aug 21 01:03:45.159008 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8401c5c6-479d-4bf0-a613-b870c12f3492 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:45.228017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: INFO dbcounter [None req-8401c5c6-479d-4bf0-a613-b870c12f3492 None None] Registered counter for database keystone Aug 21 01:03:45.248978 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG oslo_db.sqlalchemy.engines [None req-8401c5c6-479d-4bf0-a613-b870c12f3492 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81805) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 21 01:03:45.255215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writer thread running {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 21 01:03:45.376563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8401c5c6-479d-4bf0-a613-b870c12f3492 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:45.380878 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4d30c5-2bfc-41bf-a6a5-e38cc8a1d1a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:45.381497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4d30c5-2bfc-41bf-a6a5-e38cc8a1d1a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:45.381779 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4d30c5-2bfc-41bf-a6a5-e38cc8a1d1a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:45.382054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4d30c5-2bfc-41bf-a6a5-e38cc8a1d1a4 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:45.390106 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4d30c5-2bfc-41bf-a6a5-e38cc8a1d1a4 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:45.441016 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: WARNING keystone.common.rbac_enforcer.enforcer [None req-5e4d30c5-2bfc-41bf-a6a5-e38cc8a1d1a4 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 21 01:03:45.441300 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4d30c5-2bfc-41bf-a6a5-e38cc8a1d1a4 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:45.451630 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 2/4] 10.222.0.47 () {58 vars in 1324 bytes} [Sun Aug 21 01:03:45 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 21 01:03:45.457589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642ffdae-439f-4cf0-8b78-296a1f485bde None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:45.473257 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642ffdae-439f-4cf0-8b78-296a1f485bde None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:45.478166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81b2a2b-d125-48f0-a9c4-df3e5094dd00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:45.479818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81b2a2b-d125-48f0-a9c4-df3e5094dd00 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:45.479818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81b2a2b-d125-48f0-a9c4-df3e5094dd00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:45.479818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81b2a2b-d125-48f0-a9c4-df3e5094dd00 admin admin] PATH_INFO: `/v3/projects/ada79d28652f4a3aa0a174133f24108f` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:45.482727 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81b2a2b-d125-48f0-a9c4-df3e5094dd00 admin admin] RBAC: Authorizing `identity:get_project(project_id=ada79d28652f4a3aa0a174133f24108f)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:45.534558 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: WARNING keystone.common.rbac_enforcer.enforcer [None req-e81b2a2b-d125-48f0-a9c4-df3e5094dd00 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 21 01:03:45.534904 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81b2a2b-d125-48f0-a9c4-df3e5094dd00 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:45.539576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 3/5] 10.222.0.47 () {56 vars in 1207 bytes} [Sun Aug 21 01:03:45 2022] GET /identity/v3/projects/ada79d28652f4a3aa0a174133f24108f => generated 344 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:03:47.450559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad3eb1a-02e7-4ab5-89f5-24b0f0ffaa8c None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:47.451097 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad3eb1a-02e7-4ab5-89f5-24b0f0ffaa8c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:47.451097 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad3eb1a-02e7-4ab5-89f5-24b0f0ffaa8c None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:47.455186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 3/6] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:03:47 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:03:47.460632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed8bc7c-2cb9-4523-a8ff-2216bfe83b32 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:47.461070 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed8bc7c-2cb9-4523-a8ff-2216bfe83b32 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:47.461162 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed8bc7c-2cb9-4523-a8ff-2216bfe83b32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:47.481231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-5ed8bc7c-2cb9-4523-a8ff-2216bfe83b32 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:03:47.482727 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-5ed8bc7c-2cb9-4523-a8ff-2216bfe83b32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:03:47.490492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 4/7] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:03:47 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 21 01:03:47.753172 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad3eb1a-02e7-4ab5-89f5-24b0f0ffaa8c None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:47.765670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad3eb1a-02e7-4ab5-89f5-24b0f0ffaa8c None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:47.769775 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65657fc-2ba3-4788-b747-d2df240c6eb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:47.770360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65657fc-2ba3-4788-b747-d2df240c6eb6 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:47.770638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65657fc-2ba3-4788-b747-d2df240c6eb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:47.770909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65657fc-2ba3-4788-b747-d2df240c6eb6 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:47.778946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65657fc-2ba3-4788-b747-d2df240c6eb6 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:47.780729 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65657fc-2ba3-4788-b747-d2df240c6eb6 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:47.789236 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 4/8] 10.222.0.47 () {58 vars in 1324 bytes} [Sun Aug 21 01:03:47 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 21 01:03:47.794980 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed8bc7c-2cb9-4523-a8ff-2216bfe83b32 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:47.803925 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed8bc7c-2cb9-4523-a8ff-2216bfe83b32 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:47.808311 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62020267-f6e4-4aea-af06-b0c052fe86b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:47.808927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62020267-f6e4-4aea-af06-b0c052fe86b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:47.809203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62020267-f6e4-4aea-af06-b0c052fe86b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:47.809474 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62020267-f6e4-4aea-af06-b0c052fe86b8 admin admin] PATH_INFO: `/v3/users/b3234aad965e427281c9352b840a0824` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:47.810800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62020267-f6e4-4aea-af06-b0c052fe86b8 admin admin] RBAC: Authorizing `identity:get_user(user_id=b3234aad965e427281c9352b840a0824)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:47.812623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62020267-f6e4-4aea-af06-b0c052fe86b8 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:47.814146 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 5/9] 10.222.0.47 () {56 vars in 1198 bytes} [Sun Aug 21 01:03:47 2022] GET /identity/v3/users/b3234aad965e427281c9352b840a0824 => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:03:49.782244 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8dbfee-f5b6-47bf-a6c7-37ab4de73cec None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:49.782787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8dbfee-f5b6-47bf-a6c7-37ab4de73cec None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:49.782787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8dbfee-f5b6-47bf-a6c7-37ab4de73cec None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:49.784166 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 5/10] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:03:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:03:49.789548 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0d25b-33d5-431a-8661-5f1ed35f3208 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:49.789981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0d25b-33d5-431a-8661-5f1ed35f3208 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:49.790213 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0d25b-33d5-431a-8661-5f1ed35f3208 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:49.808696 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-c0a0d25b-33d5-431a-8661-5f1ed35f3208 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:03:49.809898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c0a0d25b-33d5-431a-8661-5f1ed35f3208 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:03:49.810441 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a28ddb0-db21-40ad-b481-2e1aec677831 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:49.810725 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a28ddb0-db21-40ad-b481-2e1aec677831 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:49.811000 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a28ddb0-db21-40ad-b481-2e1aec677831 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:49.812372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 6/11] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:03:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:03:49.817996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b435ff-d001-4c1b-bf19-856ce0264e3d None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:49.818275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b435ff-d001-4c1b-bf19-856ce0264e3d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:49.818545 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b435ff-d001-4c1b-bf19-856ce0264e3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:49.860807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG passlib.handlers.bcrypt [None req-09b435ff-d001-4c1b-bf19-856ce0264e3d None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=81805) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 21 01:03:49.861221 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG passlib.handlers.bcrypt [None req-09b435ff-d001-4c1b-bf19-856ce0264e3d None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81805) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 21 01:03:49.894268 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 6/12] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:03:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:03:49.908348 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-09b435ff-d001-4c1b-bf19-856ce0264e3d None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:03:49.909973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-09b435ff-d001-4c1b-bf19-856ce0264e3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:03:50.021193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 7/13] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:03:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:03:50.174933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0d25b-33d5-431a-8661-5f1ed35f3208 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:50.200973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:50.296130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b435ff-d001-4c1b-bf19-856ce0264e3d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:50.296341 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0d25b-33d5-431a-8661-5f1ed35f3208 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:50.346059 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732b6737-9b08-45bb-b054-cb109ca717b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:50.346059 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732b6737-9b08-45bb-b054-cb109ca717b9 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:50.346059 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732b6737-9b08-45bb-b054-cb109ca717b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:50.346059 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732b6737-9b08-45bb-b054-cb109ca717b9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:50.361110 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732b6737-9b08-45bb-b054-cb109ca717b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:03:50.371984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-732b6737-9b08-45bb-b054-cb109ca717b9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:03:50.371984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:03:50.371984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:03:50.371984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:03:50.371984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:03:50.371984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:03:50.371984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:03:50.371984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:03:50.371984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:03:50.371984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:03:50.371984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:03:50.371984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:03:50.371984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:03:50.371984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:03:50.371984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:03:50.371984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:03:50.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:03:50.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:03:50.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:03:50.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:03:50.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:03:50.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:03:50.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:03:50.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:03:50.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:03:50.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:03:50.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:03:50.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:03:50.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:03:50.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:03:50.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:03:50.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:03:50.374373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:03:50.374373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:03:50.374373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:03:50.374373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:03:50.374373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:03:50.374373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:03:50.374373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:03:50.374373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:03:50.374373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:03:50.374373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:03:50.374373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:03:50.374373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:03:50.374373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 7/14] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:03:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:03:50.379312 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732b6737-9b08-45bb-b054-cb109ca717b9 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:50.416952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b435ff-d001-4c1b-bf19-856ce0264e3d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:50.420225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:50.468627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1462d77-3f60-44ef-ba3f-6c21858ffe52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:50.469655 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1462d77-3f60-44ef-ba3f-6c21858ffe52 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:50.470037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1462d77-3f60-44ef-ba3f-6c21858ffe52 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:50.470420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1462d77-3f60-44ef-ba3f-6c21858ffe52 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:50.478242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:50.499837 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732b6737-9b08-45bb-b054-cb109ca717b9 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:50.537414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b058fe50-ee11-4340-9062-064ad6e5bfcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:50.538033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b058fe50-ee11-4340-9062-064ad6e5bfcd None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:50.538313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b058fe50-ee11-4340-9062-064ad6e5bfcd None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:50.538589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b058fe50-ee11-4340-9062-064ad6e5bfcd None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:50.539251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b058fe50-ee11-4340-9062-064ad6e5bfcd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:03:50.539557 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b058fe50-ee11-4340-9062-064ad6e5bfcd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:50.540989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b058fe50-ee11-4340-9062-064ad6e5bfcd None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:50.549108 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1462d77-3f60-44ef-ba3f-6c21858ffe52 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:50.550627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1462d77-3f60-44ef-ba3f-6c21858ffe52 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:50.553535 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 8/15] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:03:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:03:50.559655 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b058fe50-ee11-4340-9062-064ad6e5bfcd None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:50.657992 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 8/16] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:03:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 21 01:03:50.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b058fe50-ee11-4340-9062-064ad6e5bfcd None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:50.663580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1462d77-3f60-44ef-ba3f-6c21858ffe52 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:50.666562 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:50.699695 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9ecce8-950c-4d6a-9ed5-0a4cfa6b3a1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:50.700347 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9ecce8-950c-4d6a-9ed5-0a4cfa6b3a1c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:50.700625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9ecce8-950c-4d6a-9ed5-0a4cfa6b3a1c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:50.700890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9ecce8-950c-4d6a-9ed5-0a4cfa6b3a1c None admin] PATH_INFO: `/v3/users/b3234aad965e427281c9352b840a0824` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:50.702061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9ecce8-950c-4d6a-9ed5-0a4cfa6b3a1c None admin] RBAC: Authorizing `identity:get_user(user_id=b3234aad965e427281c9352b840a0824)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:50.703514 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9ecce8-950c-4d6a-9ed5-0a4cfa6b3a1c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:50.705139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 9/17] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:03:50 2022] GET /identity/v3/users/b3234aad965e427281c9352b840a0824 => generated 247 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:03:50.710815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9ecce8-950c-4d6a-9ed5-0a4cfa6b3a1c None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:50.725484 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:50.768942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1462d77-3f60-44ef-ba3f-6c21858ffe52 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:50.802635 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb1e17c-2f16-458c-8f5f-0e1fb46a608a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:50.803509 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb1e17c-2f16-458c-8f5f-0e1fb46a608a None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:50.803784 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb1e17c-2f16-458c-8f5f-0e1fb46a608a None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:50.804056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb1e17c-2f16-458c-8f5f-0e1fb46a608a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:50.812487 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9ecce8-950c-4d6a-9ed5-0a4cfa6b3a1c None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:50.823647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb1e17c-2f16-458c-8f5f-0e1fb46a608a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:50.825638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb1e17c-2f16-458c-8f5f-0e1fb46a608a None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:50.843720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-1cb1e17c-2f16-458c-8f5f-0e1fb46a608a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:03:50.843720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:03:50.843720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:03:50.843720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:03:50.843720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:03:50.843720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:03:50.843720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:03:50.843720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:03:50.843720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:03:50.843720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:03:50.843720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:03:50.843720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:03:50.843720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:03:50.843720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:03:50.843720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:03:50.843720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:03:50.844950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:03:50.845966 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:03:50.845966 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:03:50.845966 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:03:50.845966 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 9/18] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:03:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:03:50.848712 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623fac84-b833-4572-923f-de78f8c2b504 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:50.849354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623fac84-b833-4572-923f-de78f8c2b504 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:50.849632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623fac84-b833-4572-923f-de78f8c2b504 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:50.849902 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623fac84-b833-4572-923f-de78f8c2b504 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:50.850694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb1e17c-2f16-458c-8f5f-0e1fb46a608a None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:50.851022 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623fac84-b833-4572-923f-de78f8c2b504 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:50.852499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623fac84-b833-4572-923f-de78f8c2b504 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:50.854064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 10/19] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:03:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:03:50.859795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623fac84-b833-4572-923f-de78f8c2b504 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:50.910797 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:50.946921 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:50.959519 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb1e17c-2f16-458c-8f5f-0e1fb46a608a None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:50.968090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623fac84-b833-4572-923f-de78f8c2b504 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:50.994581 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3327a15b-04c2-4098-96c8-aa7fc5d8cd12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:50.995112 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3327a15b-04c2-4098-96c8-aa7fc5d8cd12 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:50.995388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3327a15b-04c2-4098-96c8-aa7fc5d8cd12 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:50.995657 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3327a15b-04c2-4098-96c8-aa7fc5d8cd12 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:50.996964 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3327a15b-04c2-4098-96c8-aa7fc5d8cd12 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:03:50.997174 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3327a15b-04c2-4098-96c8-aa7fc5d8cd12 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:50.998704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3327a15b-04c2-4098-96c8-aa7fc5d8cd12 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:51.006095 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1676e6d1-20a5-40a0-9dec-f7cbd16b21bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:51.006880 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1676e6d1-20a5-40a0-9dec-f7cbd16b21bc None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:51.007176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1676e6d1-20a5-40a0-9dec-f7cbd16b21bc None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:51.007425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1676e6d1-20a5-40a0-9dec-f7cbd16b21bc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:51.008176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1676e6d1-20a5-40a0-9dec-f7cbd16b21bc None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.domain.id=default, user.id=b3234aad965e427281c9352b840a0824) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:03:51.008478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1676e6d1-20a5-40a0-9dec-f7cbd16b21bc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:51.009933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1676e6d1-20a5-40a0-9dec-f7cbd16b21bc None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:51.017537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 11/20] 10.222.0.47 () {56 vars in 1322 bytes} [Sun Aug 21 01:03:50 2022] GET /identity/v3/role_assignments?user.id=b3234aad965e427281c9352b840a0824&scope.domain.id=default&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 233 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:03:51.024989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 10/21] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:03:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:03:53.352749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08129309-b17a-4148-a33a-3a494609df50 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:53.353518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08129309-b17a-4148-a33a-3a494609df50 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:53.353952 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08129309-b17a-4148-a33a-3a494609df50 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:53.360621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 12/22] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:03:53 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:03:53.366088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9711194a-e111-404e-ba79-8b152b3b4b1c None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:53.366959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9711194a-e111-404e-ba79-8b152b3b4b1c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:53.367372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9711194a-e111-404e-ba79-8b152b3b4b1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:53.402469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-9711194a-e111-404e-ba79-8b152b3b4b1c None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:03:53.404011 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9711194a-e111-404e-ba79-8b152b3b4b1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:03:53.517431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 11/23] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:03:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:03:53.692852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d7e759-0e7a-4412-8bdc-ce89b9998a95 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:53.693502 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d7e759-0e7a-4412-8bdc-ce89b9998a95 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:53.693987 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d7e759-0e7a-4412-8bdc-ce89b9998a95 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:53.696084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 13/24] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:03:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:03:53.701781 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3433aeb0-abd3-484f-a7d1-fc6681fe7fbf None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:53.702351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3433aeb0-abd3-484f-a7d1-fc6681fe7fbf None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:53.702822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3433aeb0-abd3-484f-a7d1-fc6681fe7fbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:53.722522 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-3433aeb0-abd3-484f-a7d1-fc6681fe7fbf None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:03:53.724186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-3433aeb0-abd3-484f-a7d1-fc6681fe7fbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:03:53.782430 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:53.788096 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefc65df-20cf-4436-ab5f-9310c342bcf9 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:53.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefc65df-20cf-4436-ab5f-9310c342bcf9 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:53.789148 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefc65df-20cf-4436-ab5f-9310c342bcf9 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:53.790682 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 14/25] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:03:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:03:53.796952 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d7f200-b1fd-4f96-bed8-20f35cfa67b9 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:53.797486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d7f200-b1fd-4f96-bed8-20f35cfa67b9 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:53.817758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d7f200-b1fd-4f96-bed8-20f35cfa67b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:53.817758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-f6d7f200-b1fd-4f96-bed8-20f35cfa67b9 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:03:53.818953 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 12/26] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:03:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:03:53.819226 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-f6d7f200-b1fd-4f96-bed8-20f35cfa67b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:03:53.925187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 15/27] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:03:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:03:53.954407 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8985442d-9ec1-45e0-9b82-9c8685c355bc None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:53.954698 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8985442d-9ec1-45e0-9b82-9c8685c355bc None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:53.955134 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8985442d-9ec1-45e0-9b82-9c8685c355bc None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:53.956658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 13/28] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:03:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:03:53.962397 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2e405f-c739-44e7-8093-45291e96107d None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:53.962689 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2e405f-c739-44e7-8093-45291e96107d None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:53.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2e405f-c739-44e7-8093-45291e96107d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:53.971942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8985442d-9ec1-45e0-9b82-9c8685c355bc None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:53.984168 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-7a2e405f-c739-44e7-8093-45291e96107d None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:03:53.985374 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-7a2e405f-c739-44e7-8093-45291e96107d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:03:54.124075 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 16/29] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:03:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:03:54.155805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8985442d-9ec1-45e0-9b82-9c8685c355bc None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:54.208988 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78743a81-8c5c-4b13-90cc-7f0753f013f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:54.209576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78743a81-8c5c-4b13-90cc-7f0753f013f0 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:54.209857 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78743a81-8c5c-4b13-90cc-7f0753f013f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:54.210131 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78743a81-8c5c-4b13-90cc-7f0753f013f0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:54.212938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2e405f-c739-44e7-8093-45291e96107d None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:54.217670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:54.228385 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78743a81-8c5c-4b13-90cc-7f0753f013f0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:03:54.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-78743a81-8c5c-4b13-90cc-7f0753f013f0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:03:54.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:03:54.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:03:54.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:03:54.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:03:54.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:03:54.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:03:54.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:03:54.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:03:54.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:03:54.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:03:54.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:03:54.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:03:54.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:03:54.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:03:54.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:03:54.236453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:03:54.236453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:03:54.236453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:03:54.236453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:03:54.236453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:03:54.236453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:03:54.236453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:03:54.236453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:03:54.236453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:03:54.236453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:03:54.236453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:03:54.236453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:03:54.236453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:03:54.236453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:03:54.236453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:03:54.236453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:03:54.237216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:03:54.237216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:03:54.237216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:03:54.237216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:03:54.237216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:03:54.237216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:03:54.237216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:03:54.237216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:03:54.237216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:03:54.237216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:03:54.237216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:03:54.237216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:03:54.241075 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 14/30] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:03:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:03:54.264331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78743a81-8c5c-4b13-90cc-7f0753f013f0 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:54.335913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:54.360465 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2e405f-c739-44e7-8093-45291e96107d None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:54.416055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2143b40f-91e1-476f-bb89-33519458fc08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:54.416699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2143b40f-91e1-476f-bb89-33519458fc08 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:54.416973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2143b40f-91e1-476f-bb89-33519458fc08 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:54.418087 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2143b40f-91e1-476f-bb89-33519458fc08 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:54.418866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2143b40f-91e1-476f-bb89-33519458fc08 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:54.420301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2143b40f-91e1-476f-bb89-33519458fc08 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:54.440525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78743a81-8c5c-4b13-90cc-7f0753f013f0 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:54.477796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1ddc73-9b4a-4218-91b4-e5659f7b418e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:54.478331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1ddc73-9b4a-4218-91b4-e5659f7b418e None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:54.478607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1ddc73-9b4a-4218-91b4-e5659f7b418e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:54.478879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1ddc73-9b4a-4218-91b4-e5659f7b418e None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:54.479540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1ddc73-9b4a-4218-91b4-e5659f7b418e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:03:54.480539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1ddc73-9b4a-4218-91b4-e5659f7b418e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:54.481985 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1ddc73-9b4a-4218-91b4-e5659f7b418e None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:54.498316 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 15/31] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:03:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:03:54.499220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1ddc73-9b4a-4218-91b4-e5659f7b418e None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:54.623917 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:54.628235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1ddc73-9b4a-4218-91b4-e5659f7b418e None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:54.649350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 17/32] 10.222.0.47 () {60 vars in 1131 bytes} [Sun Aug 21 01:03:54 2022] POST /identity/v3/roles => generated 219 bytes in 437 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:03:54.650425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2143b40f-91e1-476f-bb89-33519458fc08 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:54.681246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84898783-7ffa-4c27-9acb-9efb8342c93d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:54.682353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84898783-7ffa-4c27-9acb-9efb8342c93d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:54.683476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84898783-7ffa-4c27-9acb-9efb8342c93d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:54.683995 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84898783-7ffa-4c27-9acb-9efb8342c93d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:54.697653 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84898783-7ffa-4c27-9acb-9efb8342c93d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:54.699878 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84898783-7ffa-4c27-9acb-9efb8342c93d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:54.724078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-84898783-7ffa-4c27-9acb-9efb8342c93d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:03:54.724078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:03:54.724078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:03:54.724078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:03:54.724078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:03:54.724078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:03:54.724078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:03:54.724078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:03:54.724078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:03:54.724078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:03:54.724078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:03:54.724078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:03:54.724078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:03:54.724078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:03:54.724078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:03:54.724078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:03:54.725073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:03:54.725920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:03:54.725920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:03:54.725920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:03:54.728083 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:54.735939 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 16/33] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:03:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:03:54.737801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84898783-7ffa-4c27-9acb-9efb8342c93d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:54.783432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2143b40f-91e1-476f-bb89-33519458fc08 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:54.836754 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91a9c28-1fbf-409f-b555-5729477d5144 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:54.837533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91a9c28-1fbf-409f-b555-5729477d5144 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:54.837817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91a9c28-1fbf-409f-b555-5729477d5144 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:54.838204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91a9c28-1fbf-409f-b555-5729477d5144 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:54.838977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91a9c28-1fbf-409f-b555-5729477d5144 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:54.840949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91a9c28-1fbf-409f-b555-5729477d5144 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:54.869366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84898783-7ffa-4c27-9acb-9efb8342c93d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:54.909761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3a3e8f-70be-492c-acd4-72868a0d2fa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:54.910407 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3a3e8f-70be-492c-acd4-72868a0d2fa7 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:54.910696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3a3e8f-70be-492c-acd4-72868a0d2fa7 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:54.910980 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3a3e8f-70be-492c-acd4-72868a0d2fa7 None admin] PATH_INFO: `/v3/users/b3234aad965e427281c9352b840a0824` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:54.912610 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3a3e8f-70be-492c-acd4-72868a0d2fa7 None admin] RBAC: Authorizing `identity:get_user(user_id=b3234aad965e427281c9352b840a0824)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:54.914371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3a3e8f-70be-492c-acd4-72868a0d2fa7 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:54.915381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 18/34] 10.222.0.47 () {60 vars in 1131 bytes} [Sun Aug 21 01:03:54 2022] POST /identity/v3/roles => generated 225 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:03:54.916260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 17/35] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:03:54 2022] GET /identity/v3/users/b3234aad965e427281c9352b840a0824 => generated 247 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:03:54.916966 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edd351d-4bcb-43dd-80bb-4c694e1781c9 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:54.918022 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3a3e8f-70be-492c-acd4-72868a0d2fa7 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:54.922545 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edd351d-4bcb-43dd-80bb-4c694e1781c9 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:54.922890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edd351d-4bcb-43dd-80bb-4c694e1781c9 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:54.924989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 19/36] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:03:54 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:03:54.926458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edd351d-4bcb-43dd-80bb-4c694e1781c9 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:54.949459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:55.059267 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3a3e8f-70be-492c-acd4-72868a0d2fa7 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:55.075924 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edd351d-4bcb-43dd-80bb-4c694e1781c9 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:55.099826 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d986404-06a0-4db1-9f6b-81c19f7e9fe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:55.100447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d986404-06a0-4db1-9f6b-81c19f7e9fe0 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:55.100729 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d986404-06a0-4db1-9f6b-81c19f7e9fe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:55.101007 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d986404-06a0-4db1-9f6b-81c19f7e9fe0 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:55.101664 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d986404-06a0-4db1-9f6b-81c19f7e9fe0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:03:55.101969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d986404-06a0-4db1-9f6b-81c19f7e9fe0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:55.103353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d986404-06a0-4db1-9f6b-81c19f7e9fe0 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:55.105781 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:55.117247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 18/37] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:03:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:03:55.118845 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fe0843-28ce-4b91-acf2-c4ca26ca831e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:55.119144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fe0843-28ce-4b91-acf2-c4ca26ca831e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:55.119426 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fe0843-28ce-4b91-acf2-c4ca26ca831e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:55.124500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c4422e-0b9d-4cf2-b7b8-59a389d63411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:55.125153 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c4422e-0b9d-4cf2-b7b8-59a389d63411 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:55.125430 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c4422e-0b9d-4cf2-b7b8-59a389d63411 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:55.125706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c4422e-0b9d-4cf2-b7b8-59a389d63411 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:55.128731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c4422e-0b9d-4cf2-b7b8-59a389d63411 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:55.131137 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c4422e-0b9d-4cf2-b7b8-59a389d63411 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:55.131838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 20/38] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:03:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:03:55.132600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c4422e-0b9d-4cf2-b7b8-59a389d63411 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:55.151831 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-05fe0843-28ce-4b91-acf2-c4ca26ca831e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:03:55.153328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-05fe0843-28ce-4b91-acf2-c4ca26ca831e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:03:55.226524 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 19/39] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:03:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:03:55.227350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fe0843-28ce-4b91-acf2-c4ca26ca831e None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:55.248408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:55.249431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c4422e-0b9d-4cf2-b7b8-59a389d63411 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:55.283868 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649e54d8-04b2-4e1e-a687-53613d001fa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:55.284542 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649e54d8-04b2-4e1e-a687-53613d001fa1 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:55.284807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649e54d8-04b2-4e1e-a687-53613d001fa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:55.285147 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649e54d8-04b2-4e1e-a687-53613d001fa1 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:55.285739 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649e54d8-04b2-4e1e-a687-53613d001fa1 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:55.287205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649e54d8-04b2-4e1e-a687-53613d001fa1 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:55.353411 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fe0843-28ce-4b91-acf2-c4ca26ca831e None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:55.390025 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8892f8d-6fc2-40c6-ba5f-9a09853902e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:55.390693 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8892f8d-6fc2-40c6-ba5f-9a09853902e0 None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:55.390974 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8892f8d-6fc2-40c6-ba5f-9a09853902e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:55.391254 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8892f8d-6fc2-40c6-ba5f-9a09853902e0 None admin] PATH_INFO: `/v3/domains/default/users/b3234aad965e427281c9352b840a0824/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:55.393339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8892f8d-6fc2-40c6-ba5f-9a09853902e0 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=b3234aad965e427281c9352b840a0824, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:55.394786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8892f8d-6fc2-40c6-ba5f-9a09853902e0 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:55.399020 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 21/40] 10.222.0.47 () {60 vars in 1141 bytes} [Sun Aug 21 01:03:55 2022] POST /identity/v3/projects => generated 301 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:03:55.400919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0187f049-3a73-45e2-9f4d-12b1a33c0ca2 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:55.401223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0187f049-3a73-45e2-9f4d-12b1a33c0ca2 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:55.401538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0187f049-3a73-45e2-9f4d-12b1a33c0ca2 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:55.404155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 22/41] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:03:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:03:55.408824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfa4116-b4ed-464e-b4f0-b84df0c9babe None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:55.409370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfa4116-b4ed-464e-b4f0-b84df0c9babe None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:55.410472 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfa4116-b4ed-464e-b4f0-b84df0c9babe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:55.439131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-ccfa4116-b4ed-464e-b4f0-b84df0c9babe None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:03:55.440763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ccfa4116-b4ed-464e-b4f0-b84df0c9babe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:03:55.501945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:55.536164 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 23/42] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:03:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:03:55.543678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfa4116-b4ed-464e-b4f0-b84df0c9babe None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:55.595198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 20/43] 10.222.0.47 () {58 vars in 1361 bytes} [Sun Aug 21 01:03:55 2022] PUT /identity/v3/domains/default/users/b3234aad965e427281c9352b840a0824/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:03:55.689891 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfa4116-b4ed-464e-b4f0-b84df0c9babe None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:55.739882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec29858-c306-44d5-9407-b2afd25b84e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:55.740828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec29858-c306-44d5-9407-b2afd25b84e0 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:55.741250 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec29858-c306-44d5-9407-b2afd25b84e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:55.741691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec29858-c306-44d5-9407-b2afd25b84e0 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:55.742757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec29858-c306-44d5-9407-b2afd25b84e0 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:55.745199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec29858-c306-44d5-9407-b2afd25b84e0 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:55.819816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8892f8d-6fc2-40c6-ba5f-9a09853902e0 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:55.827078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 24/44] 10.222.0.47 () {60 vars in 1131 bytes} [Sun Aug 21 01:03:55 2022] POST /identity/v3/roles => generated 223 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:03:55.899755 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:56.063692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8892f8d-6fc2-40c6-ba5f-9a09853902e0 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:56.108288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d4f137-4886-4045-bc6d-90814a8c16b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:56.108936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d4f137-4886-4045-bc6d-90814a8c16b8 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:56.109266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d4f137-4886-4045-bc6d-90814a8c16b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:56.109526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d4f137-4886-4045-bc6d-90814a8c16b8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:56.110733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d4f137-4886-4045-bc6d-90814a8c16b8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:56.112402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d4f137-4886-4045-bc6d-90814a8c16b8 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:56.116816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 21/45] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:03:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:03:56.122688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec29858-c306-44d5-9407-b2afd25b84e0 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:56.164407 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:56.232513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec29858-c306-44d5-9407-b2afd25b84e0 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:56.268145 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2094fce-578d-46d8-89c2-582c97896ed6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:56.268749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2094fce-578d-46d8-89c2-582c97896ed6 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:56.269023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2094fce-578d-46d8-89c2-582c97896ed6 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:56.269293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2094fce-578d-46d8-89c2-582c97896ed6 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:56.269940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2094fce-578d-46d8-89c2-582c97896ed6 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:56.271373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2094fce-578d-46d8-89c2-582c97896ed6 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:56.426164 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 25/46] 10.222.0.47 () {60 vars in 1141 bytes} [Sun Aug 21 01:03:56 2022] POST /identity/v3/projects => generated 312 bytes in 304 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:03:57.541489 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b85e42-f756-4504-ae48-97aa35fd108a None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:57.542017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b85e42-f756-4504-ae48-97aa35fd108a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:57.542120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b85e42-f756-4504-ae48-97aa35fd108a None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:57.543421 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 22/47] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:03:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:03:57.548683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129df88c-af4f-453a-a04f-b95b52897e68 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:57.549101 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129df88c-af4f-453a-a04f-b95b52897e68 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:57.549314 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129df88c-af4f-453a-a04f-b95b52897e68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:57.673420 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-129df88c-af4f-453a-a04f-b95b52897e68 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:03:57.674536 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-129df88c-af4f-453a-a04f-b95b52897e68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:03:57.739646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 26/48] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:03:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:03:58.005354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b85e42-f756-4504-ae48-97aa35fd108a None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:58.107614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b85e42-f756-4504-ae48-97aa35fd108a None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:58.140513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c820415-df6c-4ae6-a512-8e76aced642f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:58.141138 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c820415-df6c-4ae6-a512-8e76aced642f None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:58.141417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c820415-df6c-4ae6-a512-8e76aced642f None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:58.141696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c820415-df6c-4ae6-a512-8e76aced642f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:58.144346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:58.153314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c820415-df6c-4ae6-a512-8e76aced642f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:03:58.158485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-0c820415-df6c-4ae6-a512-8e76aced642f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:03:58.158485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:03:58.158485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:03:58.158485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:03:58.158485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:03:58.158485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:03:58.158485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:03:58.158485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:03:58.158485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:03:58.158485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:03:58.158485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:03:58.158485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:03:58.158485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:03:58.158485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:03:58.158485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:03:58.158485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:03:58.162045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:03:58.162045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:03:58.162045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:03:58.162045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:03:58.162045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:03:58.162045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:03:58.162045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:03:58.162045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:03:58.162045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:03:58.162045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:03:58.162045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:03:58.162045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:03:58.162045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:03:58.162045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:03:58.162045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:03:58.162045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:03:58.162806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:03:58.162806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:03:58.162806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:03:58.162806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:03:58.162806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:03:58.162806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:03:58.162806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:03:58.162806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:03:58.162806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:03:58.162806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:03:58.162806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:03:58.162806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:03:58.162806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 23/49] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:03:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:03:58.165428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129df88c-af4f-453a-a04f-b95b52897e68 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:58.183698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:58.264756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129df88c-af4f-453a-a04f-b95b52897e68 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:58.298353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127b4f25-606a-42fb-b538-235be507b97d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:58.299011 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127b4f25-606a-42fb-b538-235be507b97d None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:58.299289 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127b4f25-606a-42fb-b538-235be507b97d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:58.299560 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127b4f25-606a-42fb-b538-235be507b97d None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:58.300252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127b4f25-606a-42fb-b538-235be507b97d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:03:58.300557 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127b4f25-606a-42fb-b538-235be507b97d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:58.301918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127b4f25-606a-42fb-b538-235be507b97d None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:58.309994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 27/50] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:03:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:03:58.314514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ccfbd1-30d4-4562-a86d-cb769001097a None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:58.314853 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ccfbd1-30d4-4562-a86d-cb769001097a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:58.315133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ccfbd1-30d4-4562-a86d-cb769001097a None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:58.315258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127b4f25-606a-42fb-b538-235be507b97d None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:58.316625 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 24/51] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:03:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:03:58.321741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78576cd9-2ca2-475a-ab98-11be3526a2c8 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:58.322019 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78576cd9-2ca2-475a-ab98-11be3526a2c8 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:58.322291 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78576cd9-2ca2-475a-ab98-11be3526a2c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:58.340681 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-78576cd9-2ca2-475a-ab98-11be3526a2c8 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:03:58.342208 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-78576cd9-2ca2-475a-ab98-11be3526a2c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:03:58.395595 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:58.409300 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 25/52] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:03:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:03:58.417334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127b4f25-606a-42fb-b538-235be507b97d None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:58.451959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666a32c9-b5c4-4038-9f40-e3f5d9ebeb1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:58.452615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666a32c9-b5c4-4038-9f40-e3f5d9ebeb1d None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:58.452894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666a32c9-b5c4-4038-9f40-e3f5d9ebeb1d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:58.453161 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666a32c9-b5c4-4038-9f40-e3f5d9ebeb1d None admin] PATH_INFO: `/v3/users/b3234aad965e427281c9352b840a0824` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:58.454291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666a32c9-b5c4-4038-9f40-e3f5d9ebeb1d None admin] RBAC: Authorizing `identity:get_user(user_id=b3234aad965e427281c9352b840a0824)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:58.455705 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666a32c9-b5c4-4038-9f40-e3f5d9ebeb1d None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:58.457517 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 28/53] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:03:58 2022] GET /identity/v3/users/b3234aad965e427281c9352b840a0824 => generated 247 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:03:58.462842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78576cd9-2ca2-475a-ab98-11be3526a2c8 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:58.559863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78576cd9-2ca2-475a-ab98-11be3526a2c8 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:58.585420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:58.594212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54ac1e2-cb89-49e7-b42b-8ad886e436e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:58.596785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54ac1e2-cb89-49e7-b42b-8ad886e436e9 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:58.596785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54ac1e2-cb89-49e7-b42b-8ad886e436e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:58.596785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54ac1e2-cb89-49e7-b42b-8ad886e436e9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:58.596785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54ac1e2-cb89-49e7-b42b-8ad886e436e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:58.598046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54ac1e2-cb89-49e7-b42b-8ad886e436e9 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:58.599582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 26/54] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:03:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:03:58.605349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666a32c9-b5c4-4038-9f40-e3f5d9ebeb1d None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:58.675360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54ac1e2-cb89-49e7-b42b-8ad886e436e9 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:58.702964 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666a32c9-b5c4-4038-9f40-e3f5d9ebeb1d None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:58.735636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97ead36-2d45-48bb-94a8-0ec3edbdc2c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:58.736266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97ead36-2d45-48bb-94a8-0ec3edbdc2c8 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:58.736541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97ead36-2d45-48bb-94a8-0ec3edbdc2c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:58.736811 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97ead36-2d45-48bb-94a8-0ec3edbdc2c8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:58.737527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97ead36-2d45-48bb-94a8-0ec3edbdc2c8 None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.domain.id=default, user.id=b3234aad965e427281c9352b840a0824) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:03:58.737846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97ead36-2d45-48bb-94a8-0ec3edbdc2c8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:58.739277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97ead36-2d45-48bb-94a8-0ec3edbdc2c8 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:58.744940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 29/55] 10.222.0.47 () {56 vars in 1322 bytes} [Sun Aug 21 01:03:58 2022] GET /identity/v3/role_assignments?user.id=b3234aad965e427281c9352b840a0824&scope.domain.id=default&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 530 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:03:58.778008 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54ac1e2-cb89-49e7-b42b-8ad886e436e9 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:58.815045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de476517-7ead-4045-aa4e-55b6c5004a3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:58.815612 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de476517-7ead-4045-aa4e-55b6c5004a3c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:58.815935 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de476517-7ead-4045-aa4e-55b6c5004a3c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:58.816307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de476517-7ead-4045-aa4e-55b6c5004a3c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:58.817470 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de476517-7ead-4045-aa4e-55b6c5004a3c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:58.818935 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de476517-7ead-4045-aa4e-55b6c5004a3c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:58.820926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 27/56] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:03:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:03:58.827355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97ead36-2d45-48bb-94a8-0ec3edbdc2c8 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:03:58.840704 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:03:58.941925 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97ead36-2d45-48bb-94a8-0ec3edbdc2c8 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:03:58.979891 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d0851b-4386-42f7-acd6-ba6acfd2d474 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:03:58.980536 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d0851b-4386-42f7-acd6-ba6acfd2d474 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:03:58.980812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d0851b-4386-42f7-acd6-ba6acfd2d474 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:03:58.981082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d0851b-4386-42f7-acd6-ba6acfd2d474 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:03:58.981727 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d0851b-4386-42f7-acd6-ba6acfd2d474 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:03:58.983170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d0851b-4386-42f7-acd6-ba6acfd2d474 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:03:59.076409 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 30/57] 10.222.0.47 () {60 vars in 1140 bytes} [Sun Aug 21 01:03:58 2022] POST /identity/v3/projects => generated 298 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:04:00.985576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b42df9-8c18-4c38-b29a-a14da739e968 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:00.986097 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b42df9-8c18-4c38-b29a-a14da739e968 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:00.986097 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b42df9-8c18-4c38-b29a-a14da739e968 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:00.987460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 28/58] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:00.992683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc463592-a020-45b6-921e-f708b76aa1d5 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:00.993052 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc463592-a020-45b6-921e-f708b76aa1d5 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:00.993225 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc463592-a020-45b6-921e-f708b76aa1d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:01.011585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-cc463592-a020-45b6-921e-f708b76aa1d5 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:01.012780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-cc463592-a020-45b6-921e-f708b76aa1d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:01.068260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:01.078306 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 31/59] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:01.344337 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b42df9-8c18-4c38-b29a-a14da739e968 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:01.405744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:01.443594 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b42df9-8c18-4c38-b29a-a14da739e968 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:01.476701 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1183d04-65ec-4f35-966d-4445097c595b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:01.477322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1183d04-65ec-4f35-966d-4445097c595b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:01.477598 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1183d04-65ec-4f35-966d-4445097c595b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:01.477866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1183d04-65ec-4f35-966d-4445097c595b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:01.478958 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1183d04-65ec-4f35-966d-4445097c595b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:01.480445 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1183d04-65ec-4f35-966d-4445097c595b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:01.481998 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 29/60] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:04:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:01.487516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc463592-a020-45b6-921e-f708b76aa1d5 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:01.584591 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc463592-a020-45b6-921e-f708b76aa1d5 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:01.619301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb17177-4fe0-409c-ba77-c249100b122e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:01.619861 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb17177-4fe0-409c-ba77-c249100b122e None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:01.620168 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb17177-4fe0-409c-ba77-c249100b122e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:01.620443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb17177-4fe0-409c-ba77-c249100b122e None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:01.621090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb17177-4fe0-409c-ba77-c249100b122e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:01.622521 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb17177-4fe0-409c-ba77-c249100b122e None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:01.715902 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 32/61] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:04:01 2022] POST /identity/v3/users => generated 275 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:04:04.624491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b627473c-c02f-4b14-9fe6-1dfdbe75ff34 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:04.625001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b627473c-c02f-4b14-9fe6-1dfdbe75ff34 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:04.625001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b627473c-c02f-4b14-9fe6-1dfdbe75ff34 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:04.626302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 30/62] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:04.642456 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b915fe-8c55-45a6-aa65-7e14e8ba9da2 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:04.642952 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b915fe-8c55-45a6-aa65-7e14e8ba9da2 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:04.643021 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b915fe-8c55-45a6-aa65-7e14e8ba9da2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:04.678286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-88b915fe-8c55-45a6-aa65-7e14e8ba9da2 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:04.679507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-88b915fe-8c55-45a6-aa65-7e14e8ba9da2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:04.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:04.768905 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 33/63] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:04.942796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7703fb77-07dd-41f3-a688-55d78ce55cfe None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:04.943784 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7703fb77-07dd-41f3-a688-55d78ce55cfe None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:04.944522 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7703fb77-07dd-41f3-a688-55d78ce55cfe None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:04.947149 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 31/64] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:04 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:04.953077 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e633b23c-8854-4f67-83bd-9a180c96b036 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:04.953359 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e633b23c-8854-4f67-83bd-9a180c96b036 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:04.953638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e633b23c-8854-4f67-83bd-9a180c96b036 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:04.973938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-e633b23c-8854-4f67-83bd-9a180c96b036 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:04.975131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e633b23c-8854-4f67-83bd-9a180c96b036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:05.044214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03b35ba-8e4b-44e1-b52e-18d283559750 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:05.044837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03b35ba-8e4b-44e1-b52e-18d283559750 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:05.045366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03b35ba-8e4b-44e1-b52e-18d283559750 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:05.047392 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 32/65] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:05.059721 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2869f917-f059-49b3-808b-018a0f3c2a16 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:05.060619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2869f917-f059-49b3-808b-018a0f3c2a16 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:05.061093 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2869f917-f059-49b3-808b-018a0f3c2a16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:05.071557 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 34/66] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:04:05.099956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-2869f917-f059-49b3-808b-018a0f3c2a16 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:05.101216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-2869f917-f059-49b3-808b-018a0f3c2a16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:05.164472 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e633b23c-8854-4f67-83bd-9a180c96b036 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:05.225822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 33/67] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:04:05.298081 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e633b23c-8854-4f67-83bd-9a180c96b036 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:05.308582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:05.335339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e213cec-dc78-4b18-921f-108ed09554e5 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:05.335640 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e213cec-dc78-4b18-921f-108ed09554e5 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:05.335885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e213cec-dc78-4b18-921f-108ed09554e5 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:05.337499 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 34/68] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:04:05.340229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb7a17e-da32-4804-9b8b-278e2d000637 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:05.340229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb7a17e-da32-4804-9b8b-278e2d000637 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:05.340229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb7a17e-da32-4804-9b8b-278e2d000637 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:05.340229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb7a17e-da32-4804-9b8b-278e2d000637 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:05.342879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884aacc1-1b5f-44fc-8150-8c0b93c7df0c None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:05.343155 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884aacc1-1b5f-44fc-8150-8c0b93c7df0c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:05.343424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884aacc1-1b5f-44fc-8150-8c0b93c7df0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:05.370807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb7a17e-da32-4804-9b8b-278e2d000637 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:05.375600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-3fb7a17e-da32-4804-9b8b-278e2d000637 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:05.375600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:05.375600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:05.375600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:05.375600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:05.375600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:05.375600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:05.375600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:05.375600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:05.375600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:05.375600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:05.375600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:05.375600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:05.375600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:05.375600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:05.375600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:05.376600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:05.376600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:05.376600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:05.376600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:05.376600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:05.376600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:05.376600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:05.376600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:05.376600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:05.376600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:05.376600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:05.376600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:05.376600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:05.376600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:05.376600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:05.376600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:05.377395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:05.377395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:05.377395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:05.377395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:05.377395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:05.377395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:05.377395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:05.377395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:05.377395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:05.377395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:05.377395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:05.377395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:05.378183 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-884aacc1-1b5f-44fc-8150-8c0b93c7df0c None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:05.378758 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-884aacc1-1b5f-44fc-8150-8c0b93c7df0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:05.381010 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 35/69] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:04:05.388064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb7a17e-da32-4804-9b8b-278e2d000637 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:05.467450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:05.483491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 35/70] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:05.483491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884aacc1-1b5f-44fc-8150-8c0b93c7df0c None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:05.515783 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb7a17e-da32-4804-9b8b-278e2d000637 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:05.595805 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a145ece-0c4b-46c8-b989-b3efb02b0f7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:05.596455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a145ece-0c4b-46c8-b989-b3efb02b0f7b None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:05.596733 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a145ece-0c4b-46c8-b989-b3efb02b0f7b None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:05.597006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a145ece-0c4b-46c8-b989-b3efb02b0f7b None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:05.597671 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a145ece-0c4b-46c8-b989-b3efb02b0f7b None admin] RBAC: Adding query filter params (name=member) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:05.597971 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a145ece-0c4b-46c8-b989-b3efb02b0f7b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:05.599491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a145ece-0c4b-46c8-b989-b3efb02b0f7b None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:05.609553 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 36/71] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:05.610837 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cac4d2d-a221-4bab-8fd9-2444f33a1c62 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:05.611126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cac4d2d-a221-4bab-8fd9-2444f33a1c62 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:05.611400 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cac4d2d-a221-4bab-8fd9-2444f33a1c62 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:05.612642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 37/72] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:05.616762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884aacc1-1b5f-44fc-8150-8c0b93c7df0c None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:05.617562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cac4d2d-a221-4bab-8fd9-2444f33a1c62 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:05.652951 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:05.655594 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97bc7b5-8c0f-47ee-9500-92c24ccfa8d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:05.656266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97bc7b5-8c0f-47ee-9500-92c24ccfa8d6 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:05.656548 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97bc7b5-8c0f-47ee-9500-92c24ccfa8d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:05.656890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97bc7b5-8c0f-47ee-9500-92c24ccfa8d6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:05.669080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97bc7b5-8c0f-47ee-9500-92c24ccfa8d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:05.673981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-c97bc7b5-8c0f-47ee-9500-92c24ccfa8d6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:05.673981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:05.673981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:05.673981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:05.673981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:05.673981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:05.673981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:05.673981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:05.673981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:05.673981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:05.673981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:05.673981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:05.673981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:05.673981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:05.673981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:05.673981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:05.674818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:05.674818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:05.674818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:05.674818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:05.674818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:05.674818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:05.674818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:05.674818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:05.674818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:05.674818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:05.674818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:05.674818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:05.674818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:05.674818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:05.674818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:05.674818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:05.675587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:05.675587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:05.675587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:05.675587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:05.675587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:05.675587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:05.675587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:05.675587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:05.675587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:05.675587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:05.675587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:05.675587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:05.675587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 36/73] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:04:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:05.676286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97bc7b5-8c0f-47ee-9500-92c24ccfa8d6 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:05.736031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:05.745144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cac4d2d-a221-4bab-8fd9-2444f33a1c62 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:05.780279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3caf5b92-c3b8-41f5-9da8-8a842325901e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:05.780918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3caf5b92-c3b8-41f5-9da8-8a842325901e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:05.781197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3caf5b92-c3b8-41f5-9da8-8a842325901e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:05.781468 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3caf5b92-c3b8-41f5-9da8-8a842325901e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:05.792438 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3caf5b92-c3b8-41f5-9da8-8a842325901e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:05.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-3caf5b92-c3b8-41f5-9da8-8a842325901e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:05.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:05.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:05.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:05.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:05.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:05.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:05.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:05.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:05.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:05.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:05.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:05.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:05.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:05.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:05.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:05.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:05.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:05.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:05.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:05.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:05.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:05.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:05.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:05.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:05.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:05.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:05.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:05.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:05.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:05.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:05.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:05.799348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:05.799348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:05.799348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:05.799348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:05.799348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:05.799348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:05.799348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:05.799348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:05.799348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:05.799348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:05.799348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:05.799348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:05.799348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 38/74] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:04:05 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:05.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a245a9-aa50-429a-9730-94523e8d6e2b None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:05.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a245a9-aa50-429a-9730-94523e8d6e2b None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:05.800300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a245a9-aa50-429a-9730-94523e8d6e2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:05.801228 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97bc7b5-8c0f-47ee-9500-92c24ccfa8d6 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:05.818856 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-f0a245a9-aa50-429a-9730-94523e8d6e2b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:05.820354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-f0a245a9-aa50-429a-9730-94523e8d6e2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:05.839544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c810707a-4bcc-4825-8a85-a563539f4a4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:05.840249 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c810707a-4bcc-4825-8a85-a563539f4a4b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:05.840529 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c810707a-4bcc-4825-8a85-a563539f4a4b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:05.840797 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c810707a-4bcc-4825-8a85-a563539f4a4b None admin] PATH_INFO: `/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:05.856637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c810707a-4bcc-4825-8a85-a563539f4a4b None admin] RBAC: Authorizing `identity:get_user(user_id=4e36c8bcb5234cd29a8d01bb6f6d6d38)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:05.858099 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c810707a-4bcc-4825-8a85-a563539f4a4b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:05.859782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 37/75] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:05 2022] GET /identity/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38 => generated 275 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:05.863460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c810707a-4bcc-4825-8a85-a563539f4a4b None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:05.891329 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 39/76] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:05.892236 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a245a9-aa50-429a-9730-94523e8d6e2b None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:05.913714 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:05.983725 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c810707a-4bcc-4825-8a85-a563539f4a4b None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:05.993686 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:06.006213 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a245a9-aa50-429a-9730-94523e8d6e2b None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:06.020337 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30bdee8-f3eb-4965-8d21-9fd22f8e1ec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:06.020955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30bdee8-f3eb-4965-8d21-9fd22f8e1ec5 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:06.021237 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30bdee8-f3eb-4965-8d21-9fd22f8e1ec5 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:06.021513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30bdee8-f3eb-4965-8d21-9fd22f8e1ec5 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:06.022182 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30bdee8-f3eb-4965-8d21-9fd22f8e1ec5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:06.022485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30bdee8-f3eb-4965-8d21-9fd22f8e1ec5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:06.023849 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30bdee8-f3eb-4965-8d21-9fd22f8e1ec5 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:06.033474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 38/77] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:06.037754 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30bdee8-f3eb-4965-8d21-9fd22f8e1ec5 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:06.044401 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b75696-7aed-479f-b4c0-c7d6f0c8dcef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:06.045082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b75696-7aed-479f-b4c0-c7d6f0c8dcef None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:06.045361 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b75696-7aed-479f-b4c0-c7d6f0c8dcef None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:06.045593 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b75696-7aed-479f-b4c0-c7d6f0c8dcef None admin] PATH_INFO: `/v3/roles/member` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:06.056571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b75696-7aed-479f-b4c0-c7d6f0c8dcef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:06.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-f3b75696-7aed-479f-b4c0-c7d6f0c8dcef None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:06.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:06.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:06.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:06.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:06.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:06.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:06.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:06.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:06.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:06.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:06.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:06.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:06.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:06.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:06.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:06.062280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:06.062280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:06.062280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:06.062280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:06.062280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:06.062280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:06.062280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:06.062280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:06.062280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:06.062280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:06.062280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:06.062280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:06.062280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:06.062280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:06.062280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:06.062280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:06.063046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:06.063046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:06.063046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:06.063046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:06.063046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:06.063046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:06.063046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:06.063046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:06.063046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:06.063046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:06.063046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:06.063046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:06.063046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 40/78] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:06.068020 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b75696-7aed-479f-b4c0-c7d6f0c8dcef None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:06.151865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30bdee8-f3eb-4965-8d21-9fd22f8e1ec5 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:06.155287 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:06.175326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b75696-7aed-479f-b4c0-c7d6f0c8dcef None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:06.188838 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422351eb-b3a3-4c02-966f-375e455662e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:06.189451 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422351eb-b3a3-4c02-966f-375e455662e1 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:06.189732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422351eb-b3a3-4c02-966f-375e455662e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:06.190007 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422351eb-b3a3-4c02-966f-375e455662e1 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:06.190667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422351eb-b3a3-4c02-966f-375e455662e1 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:06.190969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422351eb-b3a3-4c02-966f-375e455662e1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:06.192365 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422351eb-b3a3-4c02-966f-375e455662e1 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:06.200735 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 39/79] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:04:06 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:06.201527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422351eb-b3a3-4c02-966f-375e455662e1 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:06.211834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84178fc4-1554-4bcd-8f00-d891a4075b14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:06.212467 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84178fc4-1554-4bcd-8f00-d891a4075b14 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:06.212740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84178fc4-1554-4bcd-8f00-d891a4075b14 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:06.213009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84178fc4-1554-4bcd-8f00-d891a4075b14 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:06.237048 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:06.283299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84178fc4-1554-4bcd-8f00-d891a4075b14 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:06.284741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84178fc4-1554-4bcd-8f00-d891a4075b14 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:06.306715 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422351eb-b3a3-4c02-966f-375e455662e1 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:06.342922 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f366ff0b-031f-4b67-a2d9-c5b104dd9881 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:06.343560 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f366ff0b-031f-4b67-a2d9-c5b104dd9881 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:06.343837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f366ff0b-031f-4b67-a2d9-c5b104dd9881 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:06.344116 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f366ff0b-031f-4b67-a2d9-c5b104dd9881 None admin] PATH_INFO: `/v3/users/b3234aad965e427281c9352b840a0824` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:06.345289 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f366ff0b-031f-4b67-a2d9-c5b104dd9881 None admin] RBAC: Authorizing `identity:get_user(user_id=b3234aad965e427281c9352b840a0824)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:06.346746 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f366ff0b-031f-4b67-a2d9-c5b104dd9881 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:06.348437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 40/80] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:06 2022] GET /identity/v3/users/b3234aad965e427281c9352b840a0824 => generated 247 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:06.349215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f366ff0b-031f-4b67-a2d9-c5b104dd9881 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:06.367715 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:06.387345 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 41/81] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:06.388216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84178fc4-1554-4bcd-8f00-d891a4075b14 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:06.449622 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:06.453212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f366ff0b-031f-4b67-a2d9-c5b104dd9881 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:06.494201 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84178fc4-1554-4bcd-8f00-d891a4075b14 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:06.498652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455ef845-d452-4814-a015-cea5e1628852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:06.499259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455ef845-d452-4814-a015-cea5e1628852 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:06.499537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455ef845-d452-4814-a015-cea5e1628852 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:06.499813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455ef845-d452-4814-a015-cea5e1628852 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:06.500498 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455ef845-d452-4814-a015-cea5e1628852 None admin] RBAC: Adding query filter params (name=member) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:06.500801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455ef845-d452-4814-a015-cea5e1628852 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:06.502172 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455ef845-d452-4814-a015-cea5e1628852 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:06.510982 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 41/82] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:06.513024 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455ef845-d452-4814-a015-cea5e1628852 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:06.530027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749ffca9-48c6-4358-80b6-24ce012db20b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:06.530661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749ffca9-48c6-4358-80b6-24ce012db20b None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:06.531006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749ffca9-48c6-4358-80b6-24ce012db20b None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:06.531379 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749ffca9-48c6-4358-80b6-24ce012db20b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:06.532986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749ffca9-48c6-4358-80b6-24ce012db20b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:06.535251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749ffca9-48c6-4358-80b6-24ce012db20b None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:06.537392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 42/83] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:04:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:06.537981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749ffca9-48c6-4358-80b6-24ce012db20b None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:06.586222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:06.619902 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455ef845-d452-4814-a015-cea5e1628852 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:06.640085 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749ffca9-48c6-4358-80b6-24ce012db20b None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:06.656165 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c1d025-eacf-46f3-938b-b40dc73d06d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:06.656795 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c1d025-eacf-46f3-938b-b40dc73d06d2 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:06.657082 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c1d025-eacf-46f3-938b-b40dc73d06d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:06.657348 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c1d025-eacf-46f3-938b-b40dc73d06d2 None admin] PATH_INFO: `/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:06.658491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c1d025-eacf-46f3-938b-b40dc73d06d2 None admin] RBAC: Authorizing `identity:get_user(user_id=4e36c8bcb5234cd29a8d01bb6f6d6d38)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:06.659926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c1d025-eacf-46f3-938b-b40dc73d06d2 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:06.661573 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 42/84] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:06 2022] GET /identity/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38 => generated 275 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:06.662333 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c1d025-eacf-46f3-938b-b40dc73d06d2 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:06.674357 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e864129a-53ba-4314-9f5b-2bcd6595c5d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:06.674954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e864129a-53ba-4314-9f5b-2bcd6595c5d0 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:06.675188 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e864129a-53ba-4314-9f5b-2bcd6595c5d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:06.675459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e864129a-53ba-4314-9f5b-2bcd6595c5d0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:06.692838 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:06.748585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e864129a-53ba-4314-9f5b-2bcd6595c5d0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:06.750059 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e864129a-53ba-4314-9f5b-2bcd6595c5d0 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:06.771501 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c1d025-eacf-46f3-938b-b40dc73d06d2 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:06.802248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:06.816065 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a60b90-bc41-4ce7-9645-1986cb0b3d3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:06.816955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a60b90-bc41-4ce7-9645-1986cb0b3d3c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:06.820271 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a60b90-bc41-4ce7-9645-1986cb0b3d3c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:06.820271 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a60b90-bc41-4ce7-9645-1986cb0b3d3c None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:06.820271 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a60b90-bc41-4ce7-9645-1986cb0b3d3c None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:06.820599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a60b90-bc41-4ce7-9645-1986cb0b3d3c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:06.822063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 43/85] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:06 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:06.823348 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a60b90-bc41-4ce7-9645-1986cb0b3d3c None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:06.855716 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 43/86] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:06.856585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e864129a-53ba-4314-9f5b-2bcd6595c5d0 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:06.926397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:06.930316 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a60b90-bc41-4ce7-9645-1986cb0b3d3c None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:06.957116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e864129a-53ba-4314-9f5b-2bcd6595c5d0 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:06.965390 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc72fdc8-93ca-44db-b493-d9443c3c3830 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:06.966120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc72fdc8-93ca-44db-b493-d9443c3c3830 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:06.966403 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc72fdc8-93ca-44db-b493-d9443c3c3830 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:06.966675 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc72fdc8-93ca-44db-b493-d9443c3c3830 None admin] PATH_INFO: `/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:06.967814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc72fdc8-93ca-44db-b493-d9443c3c3830 None admin] RBAC: Authorizing `identity:get_user(user_id=4e36c8bcb5234cd29a8d01bb6f6d6d38)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:06.969487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc72fdc8-93ca-44db-b493-d9443c3c3830 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:06.971095 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 44/87] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:06 2022] GET /identity/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38 => generated 275 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:06.971865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc72fdc8-93ca-44db-b493-d9443c3c3830 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:06.992686 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de6c46c-781b-403c-935c-7465ee379e86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:06.993383 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de6c46c-781b-403c-935c-7465ee379e86 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:06.993664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de6c46c-781b-403c-935c-7465ee379e86 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:06.994038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de6c46c-781b-403c-935c-7465ee379e86 None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:06.994766 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de6c46c-781b-403c-935c-7465ee379e86 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:06.996279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de6c46c-781b-403c-935c-7465ee379e86 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:07.074463 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 44/88] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:04:06 2022] POST /identity/v3/users => generated 289 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:04:07.075307 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de6c46c-781b-403c-935c-7465ee379e86 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:07.077610 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc72fdc8-93ca-44db-b493-d9443c3c3830 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:07.080378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:07.118494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c4846a-6af2-4869-8b9e-327d07f4430b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:07.119125 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c4846a-6af2-4869-8b9e-327d07f4430b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:07.119903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c4846a-6af2-4869-8b9e-327d07f4430b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:07.120216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c4846a-6af2-4869-8b9e-327d07f4430b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:07.159130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:07.195871 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de6c46c-781b-403c-935c-7465ee379e86 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:07.203854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c4846a-6af2-4869-8b9e-327d07f4430b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:07.205756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c4846a-6af2-4869-8b9e-327d07f4430b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:07.241336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c087e5d5-4247-4868-b32f-f1014cbb73b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:07.242189 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c087e5d5-4247-4868-b32f-f1014cbb73b8 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:07.242469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c087e5d5-4247-4868-b32f-f1014cbb73b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:07.242742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c087e5d5-4247-4868-b32f-f1014cbb73b8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:07.243474 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c087e5d5-4247-4868-b32f-f1014cbb73b8 None admin] RBAC: Adding query filter params (role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe, scope.project.id=63ed5323666040089b15189fade45bf4, user.id=4e36c8bcb5234cd29a8d01bb6f6d6d38) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:07.243778 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c087e5d5-4247-4868-b32f-f1014cbb73b8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:07.245869 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c087e5d5-4247-4868-b32f-f1014cbb73b8 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:07.252839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 45/89] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:07 2022] GET /identity/v3/role_assignments?user.id=4e36c8bcb5234cd29a8d01bb6f6d6d38&scope.project.id=63ed5323666040089b15189fade45bf4&role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:07.253507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c087e5d5-4247-4868-b32f-f1014cbb73b8 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:07.331697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 45/90] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:07.333007 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c4846a-6af2-4869-8b9e-327d07f4430b None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:07.360770 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:07.420396 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c087e5d5-4247-4868-b32f-f1014cbb73b8 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:07.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:07.473236 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca7d80c-da68-4cb3-9410-ef6f36ee6a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:07.473878 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca7d80c-da68-4cb3-9410-ef6f36ee6a69 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:07.474168 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca7d80c-da68-4cb3-9410-ef6f36ee6a69 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:07.474443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca7d80c-da68-4cb3-9410-ef6f36ee6a69 None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:07.475618 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca7d80c-da68-4cb3-9410-ef6f36ee6a69 None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:07.477162 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca7d80c-da68-4cb3-9410-ef6f36ee6a69 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:07.479627 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 46/91] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:07 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:07.480459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca7d80c-da68-4cb3-9410-ef6f36ee6a69 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:07.490777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c4846a-6af2-4869-8b9e-327d07f4430b None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:07.528982 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb3d34c-d381-4f34-a770-4b67bf4a78e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:07.529603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb3d34c-d381-4f34-a770-4b67bf4a78e6 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:07.529880 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb3d34c-d381-4f34-a770-4b67bf4a78e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:07.530152 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb3d34c-d381-4f34-a770-4b67bf4a78e6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:07.610858 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb3d34c-d381-4f34-a770-4b67bf4a78e6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:07.612284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb3d34c-d381-4f34-a770-4b67bf4a78e6 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:07.624381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca7d80c-da68-4cb3-9410-ef6f36ee6a69 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:07.695132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0a334b-3ad2-41c1-9938-a2f5fe2cabfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:07.695780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0a334b-3ad2-41c1-9938-a2f5fe2cabfd None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:07.696032 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0a334b-3ad2-41c1-9938-a2f5fe2cabfd None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:07.696331 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0a334b-3ad2-41c1-9938-a2f5fe2cabfd None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:07.698428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0a334b-3ad2-41c1-9938-a2f5fe2cabfd None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:07.699853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0a334b-3ad2-41c1-9938-a2f5fe2cabfd None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:07.703184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 47/92] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:07 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:07.703980 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0a334b-3ad2-41c1-9938-a2f5fe2cabfd None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:07.710785 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:07.724123 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:07.787394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 46/93] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 455 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:07.788542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb3d34c-d381-4f34-a770-4b67bf4a78e6 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:07.900432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb3d34c-d381-4f34-a770-4b67bf4a78e6 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:07.925882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0a334b-3ad2-41c1-9938-a2f5fe2cabfd None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:07.962187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54cae02-76d0-4416-b63e-9be77265f993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:07.963573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54cae02-76d0-4416-b63e-9be77265f993 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:07.964611 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54cae02-76d0-4416-b63e-9be77265f993 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:07.965496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54cae02-76d0-4416-b63e-9be77265f993 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:07.965654 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7367ce-1540-457c-83c9-fc6f7a2aab83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:07.966297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7367ce-1540-457c-83c9-fc6f7a2aab83 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:07.966613 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7367ce-1540-457c-83c9-fc6f7a2aab83 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:07.966887 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7367ce-1540-457c-83c9-fc6f7a2aab83 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:07.967662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7367ce-1540-457c-83c9-fc6f7a2aab83 None admin] RBAC: Adding query filter params (role.id=fbc5961470b74a85bde6be2171f5f08f, scope.project.id=63ed5323666040089b15189fade45bf4, user.id=4e36c8bcb5234cd29a8d01bb6f6d6d38) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:07.967969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7367ce-1540-457c-83c9-fc6f7a2aab83 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:07.968691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54cae02-76d0-4416-b63e-9be77265f993 None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=63ed5323666040089b15189fade45bf4, user.id=b3234aad965e427281c9352b840a0824) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:07.969498 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54cae02-76d0-4416-b63e-9be77265f993 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:07.969650 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7367ce-1540-457c-83c9-fc6f7a2aab83 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:07.972613 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54cae02-76d0-4416-b63e-9be77265f993 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:07.980515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 48/94] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:07 2022] GET /identity/v3/role_assignments?user.id=b3234aad965e427281c9352b840a0824&scope.project.id=63ed5323666040089b15189fade45bf4&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 259 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:07.982040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54cae02-76d0-4416-b63e-9be77265f993 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:07.992835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 47/95] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:07 2022] GET /identity/v3/role_assignments?user.id=4e36c8bcb5234cd29a8d01bb6f6d6d38&scope.project.id=63ed5323666040089b15189fade45bf4&role.id=fbc5961470b74a85bde6be2171f5f08f => generated 259 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:08.068215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:08.108219 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54cae02-76d0-4416-b63e-9be77265f993 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:08.148141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1958bf-1a1e-4284-b6a1-01c69be56917 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:08.149137 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1958bf-1a1e-4284-b6a1-01c69be56917 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:08.149616 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1958bf-1a1e-4284-b6a1-01c69be56917 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:08.150077 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1958bf-1a1e-4284-b6a1-01c69be56917 None admin] PATH_INFO: `/v3/projects/407a13bf0fbc456794954f352ca2577a` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:08.151599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1958bf-1a1e-4284-b6a1-01c69be56917 None admin] RBAC: Authorizing `identity:get_project(project_id=407a13bf0fbc456794954f352ca2577a)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:08.153290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1958bf-1a1e-4284-b6a1-01c69be56917 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:08.155253 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 49/96] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:07 2022] GET /identity/v3/projects/407a13bf0fbc456794954f352ca2577a => generated 312 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:08.161520 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7367ce-1540-457c-83c9-fc6f7a2aab83 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:08.216162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:08.304801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7367ce-1540-457c-83c9-fc6f7a2aab83 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:08.341479 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d3b8b4-c827-4a03-8439-6cff07c96db8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:08.342094 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d3b8b4-c827-4a03-8439-6cff07c96db8 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:08.342371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d3b8b4-c827-4a03-8439-6cff07c96db8 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:08.342752 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d3b8b4-c827-4a03-8439-6cff07c96db8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:08.343535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d3b8b4-c827-4a03-8439-6cff07c96db8 None admin] RBAC: Adding query filter params (role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe, scope.project.id=407a13bf0fbc456794954f352ca2577a, user.id=4e36c8bcb5234cd29a8d01bb6f6d6d38) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:08.343849 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d3b8b4-c827-4a03-8439-6cff07c96db8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:08.345331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d3b8b4-c827-4a03-8439-6cff07c96db8 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:08.351701 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 48/97] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:08 2022] GET /identity/v3/role_assignments?user.id=4e36c8bcb5234cd29a8d01bb6f6d6d38&scope.project.id=407a13bf0fbc456794954f352ca2577a&role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 259 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:10.195946 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed72269-ec20-4fc9-9145-f32c5b09bc5d None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:10.196500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed72269-ec20-4fc9-9145-f32c5b09bc5d None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:10.196801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed72269-ec20-4fc9-9145-f32c5b09bc5d None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:10.199720 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 50/98] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:10 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:10.205704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d448c21-8da2-4f2e-8307-4fdcb2e685bc None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:10.206215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d448c21-8da2-4f2e-8307-4fdcb2e685bc None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:10.206292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d448c21-8da2-4f2e-8307-4fdcb2e685bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:10.233247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-7d448c21-8da2-4f2e-8307-4fdcb2e685bc None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:10.234490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7d448c21-8da2-4f2e-8307-4fdcb2e685bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:10.264187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917bab68-8f1d-4ba8-9cd7-156d93fa253f None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:10.264612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917bab68-8f1d-4ba8-9cd7-156d93fa253f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:10.265025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917bab68-8f1d-4ba8-9cd7-156d93fa253f None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:10.267449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 51/99] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:10.274078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c56e759-0e94-4c49-b956-4e7051c647a4 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:10.274506 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c56e759-0e94-4c49-b956-4e7051c647a4 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:10.274920 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c56e759-0e94-4c49-b956-4e7051c647a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:10.318708 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-2c56e759-0e94-4c49-b956-4e7051c647a4 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:10.320751 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-2c56e759-0e94-4c49-b956-4e7051c647a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:10.337535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 49/100] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:04:10.350832 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0043ce-82aa-4c50-a5a1-ba2aa488b474 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:10.351187 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0043ce-82aa-4c50-a5a1-ba2aa488b474 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:10.351394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0043ce-82aa-4c50-a5a1-ba2aa488b474 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:10.353064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 50/101] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:10.358559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c64be6-fad9-4e9f-ac25-b8edcfd9849e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:10.359015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c64be6-fad9-4e9f-ac25-b8edcfd9849e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:10.359472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c64be6-fad9-4e9f-ac25-b8edcfd9849e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:10.410405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-45c64be6-fad9-4e9f-ac25-b8edcfd9849e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:10.411626 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-45c64be6-fad9-4e9f-ac25-b8edcfd9849e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:10.454954 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:10.503008 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 52/102] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:10.597804 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 51/103] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Aug 21 01:04:10.742841 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c56e759-0e94-4c49-b956-4e7051c647a4 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:10.864393 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c64be6-fad9-4e9f-ac25-b8edcfd9849e None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:10.865957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:10.934205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c56e759-0e94-4c49-b956-4e7051c647a4 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:11.035018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221e985b-7cea-45ef-be6d-2dd1820202f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:11.036017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221e985b-7cea-45ef-be6d-2dd1820202f2 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:11.036543 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221e985b-7cea-45ef-be6d-2dd1820202f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:11.037023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221e985b-7cea-45ef-be6d-2dd1820202f2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:11.050709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221e985b-7cea-45ef-be6d-2dd1820202f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:11.055642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-221e985b-7cea-45ef-be6d-2dd1820202f2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:11.055642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:11.055642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:11.055642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:11.055642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:11.055642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:11.055642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:11.055642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:11.055642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:11.055642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:11.055642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:11.055642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:11.055642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:11.055642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:11.055642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:11.055642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:11.057650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:11.057650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:11.057650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:11.057650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:11.057650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:11.057650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:11.057650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:11.057650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:11.057650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:11.057650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:11.057650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:11.057650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:11.057650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:11.057650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:11.057650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:11.057650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:11.058659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:11.058659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:11.058659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:11.058659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:11.058659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:11.058659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:11.058659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:11.058659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:11.058659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:11.058659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:11.058659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:11.058659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:11.058659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 53/104] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:11.058659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219cfd01-637f-4ecc-b229-953c6881bfaf None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:11.059474 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219cfd01-637f-4ecc-b229-953c6881bfaf None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:11.059474 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219cfd01-637f-4ecc-b229-953c6881bfaf None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:11.060198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 54/105] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:11.068046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a70d33e-6903-4ce7-b3ff-f8f383c85ada None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:11.068361 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a70d33e-6903-4ce7-b3ff-f8f383c85ada None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:11.068633 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a70d33e-6903-4ce7-b3ff-f8f383c85ada None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:11.075161 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c64be6-fad9-4e9f-ac25-b8edcfd9849e None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:11.088772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-7a70d33e-6903-4ce7-b3ff-f8f383c85ada None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:11.090058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-7a70d33e-6903-4ce7-b3ff-f8f383c85ada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:11.146783 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fcaf75-070b-40c9-a859-445049abec11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:11.147452 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fcaf75-070b-40c9-a859-445049abec11 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:11.147734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fcaf75-070b-40c9-a859-445049abec11 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:11.148010 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fcaf75-070b-40c9-a859-445049abec11 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:11.149375 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13fcaf75-070b-40c9-a859-445049abec11 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:11.152273 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13fcaf75-070b-40c9-a859-445049abec11 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:11.154134 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 52/106] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:04:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:11.159540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fcaf75-070b-40c9-a859-445049abec11 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:11.196348 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:11.208731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 55/107] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:11.210771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99019425-6eb3-4b04-8fa8-945477bd970f None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:11.211282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99019425-6eb3-4b04-8fa8-945477bd970f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:11.211698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99019425-6eb3-4b04-8fa8-945477bd970f None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:11.213293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 56/108] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:11.214181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99019425-6eb3-4b04-8fa8-945477bd970f None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:11.287418 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:11.313257 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fcaf75-070b-40c9-a859-445049abec11 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:11.348324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99019425-6eb3-4b04-8fa8-945477bd970f None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:11.365245 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8fe5f7-c179-4eca-b318-7840930bc745 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:11.365955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8fe5f7-c179-4eca-b318-7840930bc745 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:11.366311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8fe5f7-c179-4eca-b318-7840930bc745 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:11.366657 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8fe5f7-c179-4eca-b318-7840930bc745 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:11.367833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8fe5f7-c179-4eca-b318-7840930bc745 None admin] RBAC: Adding query filter params (name=member) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:11.368246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8fe5f7-c179-4eca-b318-7840930bc745 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:11.370571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8fe5f7-c179-4eca-b318-7840930bc745 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:11.383683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 53/109] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:11 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:11.385559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8fe5f7-c179-4eca-b318-7840930bc745 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:11.396257 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58621f4-2d0f-4404-b59a-aa09ce724112 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:11.397171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58621f4-2d0f-4404-b59a-aa09ce724112 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:11.397637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58621f4-2d0f-4404-b59a-aa09ce724112 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:11.398122 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58621f4-2d0f-4404-b59a-aa09ce724112 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:11.398948 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58621f4-2d0f-4404-b59a-aa09ce724112 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:11.401264 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58621f4-2d0f-4404-b59a-aa09ce724112 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:11.486600 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:11.504352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 57/110] 10.222.0.47 () {60 vars in 1141 bytes} [Sun Aug 21 01:04:11 2022] POST /identity/v3/projects => generated 302 bytes in 290 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:04:11.505733 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de457d66-1cf2-405b-bfbd-f6ce663f63c5 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:11.506205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de457d66-1cf2-405b-bfbd-f6ce663f63c5 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:11.506662 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de457d66-1cf2-405b-bfbd-f6ce663f63c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:11.507551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8fe5f7-c179-4eca-b318-7840930bc745 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:11.530716 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-de457d66-1cf2-405b-bfbd-f6ce663f63c5 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:11.532044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-de457d66-1cf2-405b-bfbd-f6ce663f63c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:11.560589 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecc5465-4b8b-4f29-8e25-edac1a84b537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:11.561214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecc5465-4b8b-4f29-8e25-edac1a84b537 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:11.561606 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecc5465-4b8b-4f29-8e25-edac1a84b537 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:11.561982 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecc5465-4b8b-4f29-8e25-edac1a84b537 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:11.586878 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecc5465-4b8b-4f29-8e25-edac1a84b537 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:11.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-eecc5465-4b8b-4f29-8e25-edac1a84b537 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:11.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:11.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:11.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:11.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:11.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:11.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:11.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:11.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:11.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:11.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:11.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:11.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:11.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:11.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:11.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:11.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:11.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:11.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:11.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:11.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:11.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:11.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:11.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:11.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:11.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:11.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:11.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:11.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:11.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:11.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:11.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:11.594485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:11.594485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:11.594485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:11.594485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:11.594485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:11.594485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:11.594485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:11.594485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:11.594485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:11.594485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:11.594485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:11.594485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:11.594485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 54/111] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:11 2022] GET /identity/v3/roles/reader => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:11.594485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecc5465-4b8b-4f29-8e25-edac1a84b537 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:11.640168 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 58/112] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:11.641127 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de457d66-1cf2-405b-bfbd-f6ce663f63c5 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:11.701094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:11.758690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecc5465-4b8b-4f29-8e25-edac1a84b537 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:11.796251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de457d66-1cf2-405b-bfbd-f6ce663f63c5 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:11.804077 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1103df2c-7b64-415b-96d4-6e1700ed3ee7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:11.804975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1103df2c-7b64-415b-96d4-6e1700ed3ee7 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:11.805391 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1103df2c-7b64-415b-96d4-6e1700ed3ee7 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:11.805798 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1103df2c-7b64-415b-96d4-6e1700ed3ee7 None admin] PATH_INFO: `/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:11.808490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1103df2c-7b64-415b-96d4-6e1700ed3ee7 None admin] RBAC: Authorizing `identity:get_user(user_id=4e36c8bcb5234cd29a8d01bb6f6d6d38)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:11.815748 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1103df2c-7b64-415b-96d4-6e1700ed3ee7 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:11.815748 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 55/113] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:11 2022] GET /identity/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38 => generated 275 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:11.815748 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0869370-fa90-40be-8077-e194d16e86cf None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:11.815748 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0869370-fa90-40be-8077-e194d16e86cf None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:11.815748 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0869370-fa90-40be-8077-e194d16e86cf None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:11.817019 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 56/114] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:11.821998 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0869370-fa90-40be-8077-e194d16e86cf None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:11.855903 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c776663d-f58d-4fc8-9de7-cbd9e4238ad4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:11.856539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c776663d-f58d-4fc8-9de7-cbd9e4238ad4 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:11.856822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c776663d-f58d-4fc8-9de7-cbd9e4238ad4 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:11.857102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c776663d-f58d-4fc8-9de7-cbd9e4238ad4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:11.874398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c776663d-f58d-4fc8-9de7-cbd9e4238ad4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:11.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-c776663d-f58d-4fc8-9de7-cbd9e4238ad4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:11.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:11.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:11.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:11.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:11.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:11.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:11.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:11.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:11.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:11.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:11.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:11.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:11.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:11.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:11.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:11.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:11.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:11.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:11.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:11.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:11.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:11.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:11.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:11.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:11.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:11.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:11.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:11.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:11.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:11.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:11.882454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:11.883444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:11.883444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:11.883444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:11.883444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:11.883444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:11.883444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:11.883444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:11.883444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:11.883444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:11.883444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:11.883444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:11.883444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:11.887458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:11.892185 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 59/115] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:04:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:11.893337 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c776663d-f58d-4fc8-9de7-cbd9e4238ad4 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:11.970771 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0869370-fa90-40be-8077-e194d16e86cf None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:12.037039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994a56fb-ebd6-4cf4-b176-dcb44cde0e65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:12.037964 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994a56fb-ebd6-4cf4-b176-dcb44cde0e65 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:12.038445 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994a56fb-ebd6-4cf4-b176-dcb44cde0e65 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:12.038963 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994a56fb-ebd6-4cf4-b176-dcb44cde0e65 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:12.040140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994a56fb-ebd6-4cf4-b176-dcb44cde0e65 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:12.040629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994a56fb-ebd6-4cf4-b176-dcb44cde0e65 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:12.042633 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:12.044585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994a56fb-ebd6-4cf4-b176-dcb44cde0e65 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:12.049003 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c776663d-f58d-4fc8-9de7-cbd9e4238ad4 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:12.064743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 57/116] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:11 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:12.064743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e54883-d212-4b8a-b3f8-1c5163894623 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:12.064743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e54883-d212-4b8a-b3f8-1c5163894623 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:12.064743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e54883-d212-4b8a-b3f8-1c5163894623 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:12.082321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-a2e54883-d212-4b8a-b3f8-1c5163894623 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:12.083457 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a2e54883-d212-4b8a-b3f8-1c5163894623 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:12.100962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e898a9-d5f6-4835-94f9-f4c1623c7fbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:12.100962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e898a9-d5f6-4835-94f9-f4c1623c7fbb None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:12.100962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e898a9-d5f6-4835-94f9-f4c1623c7fbb None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:12.101405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e898a9-d5f6-4835-94f9-f4c1623c7fbb None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:12.103184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e898a9-d5f6-4835-94f9-f4c1623c7fbb None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:12.106350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e898a9-d5f6-4835-94f9-f4c1623c7fbb None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:12.113707 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 60/117] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:11 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:12.113707 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e898a9-d5f6-4835-94f9-f4c1623c7fbb None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:12.186304 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 58/118] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:12.187959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e54883-d212-4b8a-b3f8-1c5163894623 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:12.192272 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:12.284271 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e898a9-d5f6-4835-94f9-f4c1623c7fbb None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:12.328917 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e54883-d212-4b8a-b3f8-1c5163894623 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:12.330416 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d262014-9876-49f6-8b67-e88260e94680 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:12.331247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d262014-9876-49f6-8b67-e88260e94680 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:12.332279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d262014-9876-49f6-8b67-e88260e94680 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:12.332725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d262014-9876-49f6-8b67-e88260e94680 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:12.333473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d262014-9876-49f6-8b67-e88260e94680 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:12.333879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d262014-9876-49f6-8b67-e88260e94680 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:12.335460 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d262014-9876-49f6-8b67-e88260e94680 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:12.349875 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 61/119] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:12.349875 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d262014-9876-49f6-8b67-e88260e94680 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:12.386828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:12.399120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d131d5-baa6-4bf9-ad16-5282be236632 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:12.399931 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d131d5-baa6-4bf9-ad16-5282be236632 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:12.400905 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d131d5-baa6-4bf9-ad16-5282be236632 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:12.401277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d131d5-baa6-4bf9-ad16-5282be236632 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:12.414996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d131d5-baa6-4bf9-ad16-5282be236632 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:12.420932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-33d131d5-baa6-4bf9-ad16-5282be236632 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:12.420932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:12.420932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:12.420932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:12.420932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:12.420932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:12.420932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:12.420932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:12.420932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:12.420932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:12.420932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:12.420932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:12.420932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:12.420932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:12.420932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:12.420932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:12.421981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:12.421981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:12.421981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:12.421981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:12.421981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:12.421981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:12.421981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:12.421981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:12.421981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:12.421981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:12.421981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:12.421981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:12.421981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:12.421981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:12.421981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:12.421981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:12.422957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:12.422957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:12.422957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:12.422957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:12.422957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:12.422957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:12.422957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:12.422957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:12.422957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:12.422957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:12.422957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:12.422957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:12.424939 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 59/120] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:04:12 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:12.430541 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d131d5-baa6-4bf9-ad16-5282be236632 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:12.490452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d262014-9876-49f6-8b67-e88260e94680 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:12.503621 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:12.537341 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537a5098-106c-4d8c-ba4c-0f3d53380d19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:12.538691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537a5098-106c-4d8c-ba4c-0f3d53380d19 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:12.538969 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537a5098-106c-4d8c-ba4c-0f3d53380d19 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:12.539282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537a5098-106c-4d8c-ba4c-0f3d53380d19 None admin] PATH_INFO: `/v3/users/a1d83d21fa9c44bd83e478aefd3c492a` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:12.551611 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d131d5-baa6-4bf9-ad16-5282be236632 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:12.554221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537a5098-106c-4d8c-ba4c-0f3d53380d19 None admin] RBAC: Authorizing `identity:get_user(user_id=a1d83d21fa9c44bd83e478aefd3c492a)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:12.555808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537a5098-106c-4d8c-ba4c-0f3d53380d19 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:12.562619 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 62/121] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:12 2022] GET /identity/v3/users/a1d83d21fa9c44bd83e478aefd3c492a => generated 289 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:12.562619 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537a5098-106c-4d8c-ba4c-0f3d53380d19 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:12.594592 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e716972-0bfc-4f33-adaa-85c7f54bf7f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:12.595541 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e716972-0bfc-4f33-adaa-85c7f54bf7f4 None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:12.596006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e716972-0bfc-4f33-adaa-85c7f54bf7f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:12.596341 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e716972-0bfc-4f33-adaa-85c7f54bf7f4 None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4/users/4e36c8bcb5234cd29a8d01bb6f6d6d38/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:12.598296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e716972-0bfc-4f33-adaa-85c7f54bf7f4 None admin] RBAC: Authorizing `identity:create_grant(project_id=63ed5323666040089b15189fade45bf4, user_id=4e36c8bcb5234cd29a8d01bb6f6d6d38, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:12.599735 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e716972-0bfc-4f33-adaa-85c7f54bf7f4 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:12.658435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:12.671364 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537a5098-106c-4d8c-ba4c-0f3d53380d19 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:12.696393 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 60/122] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:04:12 2022] PUT /identity/v3/projects/63ed5323666040089b15189fade45bf4/users/4e36c8bcb5234cd29a8d01bb6f6d6d38/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:04:12.697734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e716972-0bfc-4f33-adaa-85c7f54bf7f4 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:12.723414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0625997b-36ac-4f76-abdb-9fe935443790 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:12.724261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0625997b-36ac-4f76-abdb-9fe935443790 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:12.724611 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0625997b-36ac-4f76-abdb-9fe935443790 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:12.725026 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0625997b-36ac-4f76-abdb-9fe935443790 None admin] PATH_INFO: `/v3/users/b3234aad965e427281c9352b840a0824` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:12.726415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0625997b-36ac-4f76-abdb-9fe935443790 None admin] RBAC: Authorizing `identity:get_user(user_id=b3234aad965e427281c9352b840a0824)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:12.728053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0625997b-36ac-4f76-abdb-9fe935443790 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:12.730982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 63/123] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:12 2022] GET /identity/v3/users/b3234aad965e427281c9352b840a0824 => generated 247 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:12.731908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0625997b-36ac-4f76-abdb-9fe935443790 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:12.858954 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e716972-0bfc-4f33-adaa-85c7f54bf7f4 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:12.886711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0625997b-36ac-4f76-abdb-9fe935443790 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:12.892556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:12.912663 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebedf93-8cac-41df-9084-2e0fc37b4219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:12.913518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebedf93-8cac-41df-9084-2e0fc37b4219 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:12.913704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebedf93-8cac-41df-9084-2e0fc37b4219 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:12.913949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebedf93-8cac-41df-9084-2e0fc37b4219 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:12.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebedf93-8cac-41df-9084-2e0fc37b4219 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:12.914925 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebedf93-8cac-41df-9084-2e0fc37b4219 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:12.916865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebedf93-8cac-41df-9084-2e0fc37b4219 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:12.927028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 61/124] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:04:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:12.927944 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebedf93-8cac-41df-9084-2e0fc37b4219 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:12.934964 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfbb410-0701-4de4-85fc-a0b3d8993444 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:12.935924 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfbb410-0701-4de4-85fc-a0b3d8993444 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:12.936449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfbb410-0701-4de4-85fc-a0b3d8993444 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:12.936926 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfbb410-0701-4de4-85fc-a0b3d8993444 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:12.950905 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfbb410-0701-4de4-85fc-a0b3d8993444 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:12.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-5cfbb410-0701-4de4-85fc-a0b3d8993444 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:12.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:12.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:12.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:12.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:12.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:12.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:12.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:12.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:12.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:12.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:12.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:12.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:12.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:12.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:12.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:12.957436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:12.957436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:12.957436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:12.957436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:12.957436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:12.957436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:12.957436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:12.957436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:12.957436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:12.957436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:12.957436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:12.957436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:12.957436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:12.957436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:12.957436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:12.957436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:12.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:12.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:12.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:12.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:12.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:12.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:12.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:12.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:12.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:12.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:12.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:12.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:12.960756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 64/125] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:12 2022] GET /identity/v3/roles/member => generated 84 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:12.961923 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfbb410-0701-4de4-85fc-a0b3d8993444 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:12.980034 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:13.044333 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebedf93-8cac-41df-9084-2e0fc37b4219 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:13.080302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfbb410-0701-4de4-85fc-a0b3d8993444 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:13.087372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa93c06b-da98-4c06-9144-1b0d6d9b777e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:13.088773 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa93c06b-da98-4c06-9144-1b0d6d9b777e None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:13.089196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa93c06b-da98-4c06-9144-1b0d6d9b777e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:13.089607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa93c06b-da98-4c06-9144-1b0d6d9b777e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:13.121259 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638e78c5-6ab1-4874-874e-de006d418339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:13.122097 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638e78c5-6ab1-4874-874e-de006d418339 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:13.122415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638e78c5-6ab1-4874-874e-de006d418339 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:13.122791 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638e78c5-6ab1-4874-874e-de006d418339 None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:13.124199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638e78c5-6ab1-4874-874e-de006d418339 None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:13.126091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638e78c5-6ab1-4874-874e-de006d418339 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:13.132575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 65/126] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:12 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:13.132575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638e78c5-6ab1-4874-874e-de006d418339 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:13.160942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:13.178074 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa93c06b-da98-4c06-9144-1b0d6d9b777e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:13.181261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa93c06b-da98-4c06-9144-1b0d6d9b777e None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:13.250184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:13.271982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638e78c5-6ab1-4874-874e-de006d418339 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:13.309005 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5233b427-6361-478c-b763-e482f7330924 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:13.309819 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5233b427-6361-478c-b763-e482f7330924 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:13.310198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5233b427-6361-478c-b763-e482f7330924 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:13.310595 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5233b427-6361-478c-b763-e482f7330924 None admin] PATH_INFO: `/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:13.312024 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5233b427-6361-478c-b763-e482f7330924 None admin] RBAC: Authorizing `identity:get_user(user_id=4e36c8bcb5234cd29a8d01bb6f6d6d38)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:13.313855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5233b427-6361-478c-b763-e482f7330924 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:13.315571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 66/127] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:13 2022] GET /identity/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38 => generated 275 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:13.316544 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5233b427-6361-478c-b763-e482f7330924 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:13.325264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 62/128] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:13.326367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa93c06b-da98-4c06-9144-1b0d6d9b777e None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:13.453300 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:13.453916 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5233b427-6361-478c-b763-e482f7330924 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:13.475158 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa93c06b-da98-4c06-9144-1b0d6d9b777e None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:13.493278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4554b1ed-84a1-468d-9639-29301bf85552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:13.494115 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4554b1ed-84a1-468d-9639-29301bf85552 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:13.494590 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4554b1ed-84a1-468d-9639-29301bf85552 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:13.494969 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4554b1ed-84a1-468d-9639-29301bf85552 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:13.495843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4554b1ed-84a1-468d-9639-29301bf85552 None admin] RBAC: Adding query filter params (name=member) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:13.496279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4554b1ed-84a1-468d-9639-29301bf85552 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:13.497842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4554b1ed-84a1-468d-9639-29301bf85552 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:13.508208 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 67/129] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:13.509304 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4554b1ed-84a1-468d-9639-29301bf85552 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:13.525105 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e05963d-9d28-45e6-97bd-f026c0a66a0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:13.526653 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e05963d-9d28-45e6-97bd-f026c0a66a0d None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:13.527079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e05963d-9d28-45e6-97bd-f026c0a66a0d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:13.527825 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:13.528922 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e05963d-9d28-45e6-97bd-f026c0a66a0d None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4/users/b3234aad965e427281c9352b840a0824/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:13.531645 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e05963d-9d28-45e6-97bd-f026c0a66a0d None admin] RBAC: Authorizing `identity:create_grant(project_id=63ed5323666040089b15189fade45bf4, user_id=b3234aad965e427281c9352b840a0824, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:13.533594 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e05963d-9d28-45e6-97bd-f026c0a66a0d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:13.625489 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4554b1ed-84a1-468d-9639-29301bf85552 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:13.632573 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 63/130] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:04:13 2022] PUT /identity/v3/projects/63ed5323666040089b15189fade45bf4/users/b3234aad965e427281c9352b840a0824/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:04:13.633409 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e05963d-9d28-45e6-97bd-f026c0a66a0d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:13.676472 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718adeab-37f2-4c7b-9845-66b6c10e3eec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:13.677467 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718adeab-37f2-4c7b-9845-66b6c10e3eec None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:13.677761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718adeab-37f2-4c7b-9845-66b6c10e3eec None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:13.678038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718adeab-37f2-4c7b-9845-66b6c10e3eec None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:13.680064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718adeab-37f2-4c7b-9845-66b6c10e3eec None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:13.682267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718adeab-37f2-4c7b-9845-66b6c10e3eec None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:13.685097 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 68/131] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:13 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:13.692339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718adeab-37f2-4c7b-9845-66b6c10e3eec None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:13.800460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e05963d-9d28-45e6-97bd-f026c0a66a0d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:13.806262 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:13.831041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718adeab-37f2-4c7b-9845-66b6c10e3eec None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:13.853402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8923b08b-b581-4ac5-98dd-501c9e1a7dd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:13.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8923b08b-b581-4ac5-98dd-501c9e1a7dd5 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:13.854347 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8923b08b-b581-4ac5-98dd-501c9e1a7dd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:13.855232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8923b08b-b581-4ac5-98dd-501c9e1a7dd5 None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:13.856881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8923b08b-b581-4ac5-98dd-501c9e1a7dd5 None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:13.858625 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8923b08b-b581-4ac5-98dd-501c9e1a7dd5 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:13.861767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 64/132] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:13 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:13.863785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848e56d6-dc4a-4390-b425-ff8cf31d80fe None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:13.864078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848e56d6-dc4a-4390-b425-ff8cf31d80fe None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:13.864373 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848e56d6-dc4a-4390-b425-ff8cf31d80fe None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:13.866427 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 65/133] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:13.867445 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848e56d6-dc4a-4390-b425-ff8cf31d80fe None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:13.875717 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74acd68-ee38-4f9d-bed8-8a854103b886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:13.876876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74acd68-ee38-4f9d-bed8-8a854103b886 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:13.877395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74acd68-ee38-4f9d-bed8-8a854103b886 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:13.877880 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74acd68-ee38-4f9d-bed8-8a854103b886 None admin] PATH_INFO: `/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:13.879309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74acd68-ee38-4f9d-bed8-8a854103b886 None admin] RBAC: Authorizing `identity:get_user(user_id=4e36c8bcb5234cd29a8d01bb6f6d6d38)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:13.881120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74acd68-ee38-4f9d-bed8-8a854103b886 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:13.883056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 69/134] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:13 2022] GET /identity/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38 => generated 275 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:13.885313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bb6b52-0e5e-40cd-b06e-e5f69ba3dbd2 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:13.885802 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bb6b52-0e5e-40cd-b06e-e5f69ba3dbd2 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:13.886280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bb6b52-0e5e-40cd-b06e-e5f69ba3dbd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:13.895311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:13.906239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-60bb6b52-0e5e-40cd-b06e-e5f69ba3dbd2 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:13.907703 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-60bb6b52-0e5e-40cd-b06e-e5f69ba3dbd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:13.996025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 70/135] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:13.997322 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bb6b52-0e5e-40cd-b06e-e5f69ba3dbd2 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:13.998574 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848e56d6-dc4a-4390-b425-ff8cf31d80fe None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:14.044042 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92ee804-8d21-4008-8820-ba3995d2dee4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:14.044817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92ee804-8d21-4008-8820-ba3995d2dee4 None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:14.045160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92ee804-8d21-4008-8820-ba3995d2dee4 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:14.045782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92ee804-8d21-4008-8820-ba3995d2dee4 None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4/users/4e36c8bcb5234cd29a8d01bb6f6d6d38/roles/fbc5961470b74a85bde6be2171f5f08f` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:14.048519 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92ee804-8d21-4008-8820-ba3995d2dee4 None admin] RBAC: Authorizing `identity:create_grant(project_id=63ed5323666040089b15189fade45bf4, user_id=4e36c8bcb5234cd29a8d01bb6f6d6d38, role_id=fbc5961470b74a85bde6be2171f5f08f)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:14.050716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92ee804-8d21-4008-8820-ba3995d2dee4 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:14.089615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:14.112235 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bb6b52-0e5e-40cd-b06e-e5f69ba3dbd2 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:14.146711 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 66/136] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:04:13 2022] PUT /identity/v3/projects/63ed5323666040089b15189fade45bf4/users/4e36c8bcb5234cd29a8d01bb6f6d6d38/roles/fbc5961470b74a85bde6be2171f5f08f => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:04:14.147535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92ee804-8d21-4008-8820-ba3995d2dee4 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:14.154359 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b170138-ab10-4a81-86d3-3a7e9a5d98cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:14.155235 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b170138-ab10-4a81-86d3-3a7e9a5d98cf None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:14.155650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b170138-ab10-4a81-86d3-3a7e9a5d98cf None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:14.156168 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b170138-ab10-4a81-86d3-3a7e9a5d98cf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:14.157187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b170138-ab10-4a81-86d3-3a7e9a5d98cf None admin] RBAC: Adding query filter params (role.id=b5eaad32927541e2a79d4904b5230343, scope.project.id=63ed5323666040089b15189fade45bf4, user.id=a1d83d21fa9c44bd83e478aefd3c492a) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:14.157692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b170138-ab10-4a81-86d3-3a7e9a5d98cf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:14.159360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b170138-ab10-4a81-86d3-3a7e9a5d98cf None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:14.167241 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 71/137] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:13 2022] GET /identity/v3/role_assignments?user.id=a1d83d21fa9c44bd83e478aefd3c492a&scope.project.id=63ed5323666040089b15189fade45bf4&role.id=b5eaad32927541e2a79d4904b5230343 => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:14.287668 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:14.323494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92ee804-8d21-4008-8820-ba3995d2dee4 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:14.389497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4187a645-0a35-4fbe-b665-2edc260c90ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:14.390368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4187a645-0a35-4fbe-b665-2edc260c90ed None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:14.390750 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4187a645-0a35-4fbe-b665-2edc260c90ed None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:14.391120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4187a645-0a35-4fbe-b665-2edc260c90ed None admin] PATH_INFO: `/v3/projects/407a13bf0fbc456794954f352ca2577a` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:14.392457 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4187a645-0a35-4fbe-b665-2edc260c90ed None admin] RBAC: Authorizing `identity:get_project(project_id=407a13bf0fbc456794954f352ca2577a)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:14.397180 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4187a645-0a35-4fbe-b665-2edc260c90ed None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:14.399166 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 67/138] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:14 2022] GET /identity/v3/projects/407a13bf0fbc456794954f352ca2577a => generated 312 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:14.405497 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b170138-ab10-4a81-86d3-3a7e9a5d98cf None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:14.409575 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4187a645-0a35-4fbe-b665-2edc260c90ed None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:14.563797 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4187a645-0a35-4fbe-b665-2edc260c90ed None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:14.608916 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdce2e2-4671-42c6-82b0-98d229b5bf51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:14.609640 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdce2e2-4671-42c6-82b0-98d229b5bf51 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:14.610013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdce2e2-4671-42c6-82b0-98d229b5bf51 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:14.610380 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdce2e2-4671-42c6-82b0-98d229b5bf51 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:14.611590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdce2e2-4671-42c6-82b0-98d229b5bf51 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:14.613172 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdce2e2-4671-42c6-82b0-98d229b5bf51 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:14.615157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 68/139] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:04:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:14.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdce2e2-4671-42c6-82b0-98d229b5bf51 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:14.644184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:14.691008 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b170138-ab10-4a81-86d3-3a7e9a5d98cf None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:14.732942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a0dca8-1a87-4813-ad9d-b34b13eaca92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:14.733706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a0dca8-1a87-4813-ad9d-b34b13eaca92 None admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:14.734004 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a0dca8-1a87-4813-ad9d-b34b13eaca92 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:14.734282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a0dca8-1a87-4813-ad9d-b34b13eaca92 None admin] PATH_INFO: `/v3/projects/407a13bf0fbc456794954f352ca2577a/users/4e36c8bcb5234cd29a8d01bb6f6d6d38/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:14.741518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a0dca8-1a87-4813-ad9d-b34b13eaca92 None admin] RBAC: Authorizing `identity:create_grant(project_id=407a13bf0fbc456794954f352ca2577a, user_id=4e36c8bcb5234cd29a8d01bb6f6d6d38, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:14.743018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a0dca8-1a87-4813-ad9d-b34b13eaca92 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:14.818695 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdce2e2-4671-42c6-82b0-98d229b5bf51 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:14.827317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 72/140] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:04:14 2022] PUT /identity/v3/projects/407a13bf0fbc456794954f352ca2577a/users/4e36c8bcb5234cd29a8d01bb6f6d6d38/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:04:14.863384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05436350-0cfb-493b-b4e9-65a5859a91f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:14.863926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05436350-0cfb-493b-b4e9-65a5859a91f4 None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:14.864251 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05436350-0cfb-493b-b4e9-65a5859a91f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:14.864586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05436350-0cfb-493b-b4e9-65a5859a91f4 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:14.865247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05436350-0cfb-493b-b4e9-65a5859a91f4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:14.866724 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05436350-0cfb-493b-b4e9-65a5859a91f4 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:14.924202 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e6a9b4-c7a1-475b-89de-454b06a93b19 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:14.924573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e6a9b4-c7a1-475b-89de-454b06a93b19 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:14.924855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e6a9b4-c7a1-475b-89de-454b06a93b19 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:14.926273 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 73/141] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:04:14.932339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be88ba74-2d75-4b88-a7dd-dbfa89cf07aa None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:14.932646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be88ba74-2d75-4b88-a7dd-dbfa89cf07aa None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:14.932936 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be88ba74-2d75-4b88-a7dd-dbfa89cf07aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:14.939737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:14.953590 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-be88ba74-2d75-4b88-a7dd-dbfa89cf07aa None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:14.954873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-be88ba74-2d75-4b88-a7dd-dbfa89cf07aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:14.996165 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 69/142] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:04:14 2022] POST /identity/v3/users => generated 283 bytes in 375 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:04:15.043119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 74/143] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:04:15.381798 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05436350-0cfb-493b-b4e9-65a5859a91f4 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:15.478605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:15.523254 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05436350-0cfb-493b-b4e9-65a5859a91f4 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:15.582765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52df14e-5642-4834-8e08-626e1ace71a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:15.583518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52df14e-5642-4834-8e08-626e1ace71a3 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:15.583700 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52df14e-5642-4834-8e08-626e1ace71a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:15.583974 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52df14e-5642-4834-8e08-626e1ace71a3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:15.601073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52df14e-5642-4834-8e08-626e1ace71a3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:15.606934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-c52df14e-5642-4834-8e08-626e1ace71a3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:15.606934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:15.606934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:15.606934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:15.606934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:15.606934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:15.606934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:15.606934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:15.606934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:15.606934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:15.606934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:15.606934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:15.606934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:15.606934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:15.606934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:15.606934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:15.607807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:15.607807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:15.607807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:15.607807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:15.607807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:15.607807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:15.607807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:15.607807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:15.607807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:15.607807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:15.607807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:15.607807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:15.607807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:15.607807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:15.607807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:15.607807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:15.608596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:15.608596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:15.608596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:15.608596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:15.608596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:15.608596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:15.608596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:15.608596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:15.608596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:15.608596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:15.608596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:15.608596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:15.608596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 70/144] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:15.614112 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be88ba74-2d75-4b88-a7dd-dbfa89cf07aa None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:15.756763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be88ba74-2d75-4b88-a7dd-dbfa89cf07aa None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:15.799383 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:15.804748 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de368a7-3f2e-40c2-bb8c-221aa940e1ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:15.805434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de368a7-3f2e-40c2-bb8c-221aa940e1ca None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:15.805707 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de368a7-3f2e-40c2-bb8c-221aa940e1ca None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:15.805970 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de368a7-3f2e-40c2-bb8c-221aa940e1ca None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:15.807202 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de368a7-3f2e-40c2-bb8c-221aa940e1ca None admin] RBAC: Adding query filter params (name=member) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:15.807499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de368a7-3f2e-40c2-bb8c-221aa940e1ca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:15.808918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de368a7-3f2e-40c2-bb8c-221aa940e1ca None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:15.824302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 75/145] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:15.830248 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52df14e-5642-4834-8e08-626e1ace71a3 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:16.015312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52df14e-5642-4834-8e08-626e1ace71a3 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:16.058734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:16.075711 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cde4630-1252-4e6a-992c-75037b04b191 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:16.076835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cde4630-1252-4e6a-992c-75037b04b191 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:16.085266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cde4630-1252-4e6a-992c-75037b04b191 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:16.085266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cde4630-1252-4e6a-992c-75037b04b191 None admin] PATH_INFO: `/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:16.085266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cde4630-1252-4e6a-992c-75037b04b191 None admin] RBAC: Authorizing `identity:get_user(user_id=4e36c8bcb5234cd29a8d01bb6f6d6d38)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:16.088472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cde4630-1252-4e6a-992c-75037b04b191 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:16.092076 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 71/146] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:15 2022] GET /identity/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38 => generated 275 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:16.097849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de368a7-3f2e-40c2-bb8c-221aa940e1ca None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:16.249799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de368a7-3f2e-40c2-bb8c-221aa940e1ca None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:16.299078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe7ea2b-9bf0-4492-8090-94f594798f8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:16.299078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe7ea2b-9bf0-4492-8090-94f594798f8c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:16.299078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe7ea2b-9bf0-4492-8090-94f594798f8c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:16.299078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe7ea2b-9bf0-4492-8090-94f594798f8c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:16.336528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d7db19-d0d5-4a44-91e1-7a9294130e6f None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:16.336828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d7db19-d0d5-4a44-91e1-7a9294130e6f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:16.337087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d7db19-d0d5-4a44-91e1-7a9294130e6f None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:16.341122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 72/147] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:16.374478 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade36d2c-38c6-4e9b-a7f1-ac3408119d6a None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:16.374796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade36d2c-38c6-4e9b-a7f1-ac3408119d6a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:16.375033 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade36d2c-38c6-4e9b-a7f1-ac3408119d6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:16.388527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:16.404220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe7ea2b-9bf0-4492-8090-94f594798f8c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:16.411885 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe7ea2b-9bf0-4492-8090-94f594798f8c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:16.422176 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-ade36d2c-38c6-4e9b-a7f1-ac3408119d6a None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:16.423386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ade36d2c-38c6-4e9b-a7f1-ac3408119d6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:16.611422 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 73/148] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:16.621589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 76/149] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 524 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 21 01:04:16.628010 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade36d2c-38c6-4e9b-a7f1-ac3408119d6a None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:16.705841 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0022d5c3-f2de-4330-a07f-e796b876dcae None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:16.706442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0022d5c3-f2de-4330-a07f-e796b876dcae None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:16.706933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0022d5c3-f2de-4330-a07f-e796b876dcae None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:16.708781 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 77/150] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:16.714149 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d8ef15-a6c7-4277-8b06-624e33e39f31 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:16.714693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d8ef15-a6c7-4277-8b06-624e33e39f31 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:16.715157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d8ef15-a6c7-4277-8b06-624e33e39f31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:16.760132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-69d8ef15-a6c7-4277-8b06-624e33e39f31 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:16.761666 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-69d8ef15-a6c7-4277-8b06-624e33e39f31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:16.853550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade36d2c-38c6-4e9b-a7f1-ac3408119d6a None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:16.882490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:16.900073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c52856-10c9-416d-9310-1953d8dd5ae4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:16.900767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c52856-10c9-416d-9310-1953d8dd5ae4 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:16.901042 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c52856-10c9-416d-9310-1953d8dd5ae4 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:16.901311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c52856-10c9-416d-9310-1953d8dd5ae4 None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:16.905084 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c52856-10c9-416d-9310-1953d8dd5ae4 None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:16.906733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c52856-10c9-416d-9310-1953d8dd5ae4 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:16.908936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 74/151] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:16 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:16.914616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c52856-10c9-416d-9310-1953d8dd5ae4 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:16.942436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 78/152] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:04:17.057353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c52856-10c9-416d-9310-1953d8dd5ae4 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:17.093381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b25f24-7799-4016-808a-2f27a2705f4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:17.093990 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b25f24-7799-4016-808a-2f27a2705f4f None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:17.094263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b25f24-7799-4016-808a-2f27a2705f4f None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:17.094531 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b25f24-7799-4016-808a-2f27a2705f4f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:17.095257 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b25f24-7799-4016-808a-2f27a2705f4f None admin] RBAC: Adding query filter params (role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe, scope.project.id=63ed5323666040089b15189fade45bf4, user.id=4e36c8bcb5234cd29a8d01bb6f6d6d38) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:17.095560 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b25f24-7799-4016-808a-2f27a2705f4f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:17.097041 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b25f24-7799-4016-808a-2f27a2705f4f None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:17.103993 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 75/153] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:16 2022] GET /identity/v3/role_assignments?user.id=4e36c8bcb5234cd29a8d01bb6f6d6d38&scope.project.id=63ed5323666040089b15189fade45bf4&role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 608 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:17.411528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d8ef15-a6c7-4277-8b06-624e33e39f31 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:17.444313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:17.533031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b25f24-7799-4016-808a-2f27a2705f4f None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:17.552143 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d8ef15-a6c7-4277-8b06-624e33e39f31 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:17.620572 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:17.657432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d15b77d-d729-46d7-a4c4-caea0160b162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:17.658102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d15b77d-d729-46d7-a4c4-caea0160b162 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:17.658360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d15b77d-d729-46d7-a4c4-caea0160b162 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:17.658629 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d15b77d-d729-46d7-a4c4-caea0160b162 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:17.690886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b25f24-7799-4016-808a-2f27a2705f4f None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:17.704822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d15b77d-d729-46d7-a4c4-caea0160b162 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:17.709967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-4d15b77d-d729-46d7-a4c4-caea0160b162 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:17.709967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:17.709967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:17.709967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:17.709967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:17.709967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:17.709967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:17.709967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:17.709967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:17.709967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:17.709967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:17.709967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:17.709967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:17.709967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:17.709967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:17.709967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:17.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:17.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:17.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:17.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:17.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:17.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:17.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:17.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:17.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:17.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:17.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:17.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:17.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:17.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:17.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:17.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:17.711612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:17.711612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:17.711612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:17.711612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:17.711612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:17.711612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:17.711612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:17.711612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:17.711612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:17.711612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:17.711612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:17.711612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:17.711612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 79/154] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:04:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:17.721156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d15b77d-d729-46d7-a4c4-caea0160b162 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:17.755695 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b2cfc9-17eb-43ee-a8aa-031438f91e91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:17.756773 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b2cfc9-17eb-43ee-a8aa-031438f91e91 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:17.757267 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b2cfc9-17eb-43ee-a8aa-031438f91e91 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:17.757994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b2cfc9-17eb-43ee-a8aa-031438f91e91 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:17.789246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b2cfc9-17eb-43ee-a8aa-031438f91e91 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:17.802658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-f3b2cfc9-17eb-43ee-a8aa-031438f91e91 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:17.802658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:17.802658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:17.802658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:17.802658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:17.802658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:17.802658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:17.802658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:17.802658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:17.802658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:17.802658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:17.802658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:17.802658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:17.802658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:17.802658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:17.802658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:17.803667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:17.803667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:17.803667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:17.803667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:17.803667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:17.803667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:17.803667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:17.803667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:17.803667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:17.803667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:17.803667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:17.803667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:17.803667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:17.803667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:17.803667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:17.803667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:17.804469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:17.804469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:17.804469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:17.804469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:17.804469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:17.804469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:17.804469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:17.804469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:17.804469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:17.804469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:17.804469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:17.804469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:17.805552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 76/155] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:17 2022] GET /identity/v3/roles/reader => generated 84 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:17.812726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b2cfc9-17eb-43ee-a8aa-031438f91e91 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:18.001996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:18.046333 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d15b77d-d729-46d7-a4c4-caea0160b162 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:18.052625 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b2cfc9-17eb-43ee-a8aa-031438f91e91 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:18.080057 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:18.117365 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d10e9b0-d3e3-4043-91bb-bea2073d3d69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:18.124296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d10e9b0-d3e3-4043-91bb-bea2073d3d69 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:18.124296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d10e9b0-d3e3-4043-91bb-bea2073d3d69 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:18.124296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d10e9b0-d3e3-4043-91bb-bea2073d3d69 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:18.124296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d10e9b0-d3e3-4043-91bb-bea2073d3d69 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:18.124296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d10e9b0-d3e3-4043-91bb-bea2073d3d69 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:18.126138 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d10e9b0-d3e3-4043-91bb-bea2073d3d69 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:18.132028 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6866f11-8b04-4e86-bb4d-ef58ce5ae6fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:18.132682 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6866f11-8b04-4e86-bb4d-ef58ce5ae6fb None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:18.132961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6866f11-8b04-4e86-bb4d-ef58ce5ae6fb None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:18.133228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6866f11-8b04-4e86-bb4d-ef58ce5ae6fb None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:18.133905 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6866f11-8b04-4e86-bb4d-ef58ce5ae6fb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:18.134204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6866f11-8b04-4e86-bb4d-ef58ce5ae6fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:18.135743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6866f11-8b04-4e86-bb4d-ef58ce5ae6fb None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:18.142995 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 77/156] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:17 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 332 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:18.144599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59413692-4873-4d19-876b-1bc1ff648a12 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:18.145196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59413692-4873-4d19-876b-1bc1ff648a12 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:18.145678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59413692-4873-4d19-876b-1bc1ff648a12 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:18.147149 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 78/157] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:18.153301 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59413692-4873-4d19-876b-1bc1ff648a12 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:18.158290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 80/158] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 437 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:18.159493 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334d0f72-09e4-49fd-b6ee-2a781642c13d None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:18.159776 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334d0f72-09e4-49fd-b6ee-2a781642c13d None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:18.160049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334d0f72-09e4-49fd-b6ee-2a781642c13d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:18.196562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-334d0f72-09e4-49fd-b6ee-2a781642c13d None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:18.198395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-334d0f72-09e4-49fd-b6ee-2a781642c13d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:18.335046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 81/159] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:18.335843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334d0f72-09e4-49fd-b6ee-2a781642c13d None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:18.368711 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59413692-4873-4d19-876b-1bc1ff648a12 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:18.429187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:18.437237 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e335905-465b-46b5-ad95-672d5678d638 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:18.438231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e335905-465b-46b5-ad95-672d5678d638 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:18.438505 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e335905-465b-46b5-ad95-672d5678d638 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:18.438871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e335905-465b-46b5-ad95-672d5678d638 None admin] PATH_INFO: `/v3/users/a1d83d21fa9c44bd83e478aefd3c492a` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:18.440129 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e335905-465b-46b5-ad95-672d5678d638 None admin] RBAC: Authorizing `identity:get_user(user_id=a1d83d21fa9c44bd83e478aefd3c492a)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:18.441666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e335905-465b-46b5-ad95-672d5678d638 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:18.446389 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 79/160] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:18 2022] GET /identity/v3/users/a1d83d21fa9c44bd83e478aefd3c492a => generated 289 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:18.452415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e335905-465b-46b5-ad95-672d5678d638 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:18.456139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334d0f72-09e4-49fd-b6ee-2a781642c13d None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:18.492430 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2885fde-976a-4bf2-b639-a3a964da28ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:18.493332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2885fde-976a-4bf2-b639-a3a964da28ce None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:18.494325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2885fde-976a-4bf2-b639-a3a964da28ce None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:18.494732 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2885fde-976a-4bf2-b639-a3a964da28ce None admin] PATH_INFO: `/v3/users/b3234aad965e427281c9352b840a0824` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:18.496070 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2885fde-976a-4bf2-b639-a3a964da28ce None admin] RBAC: Authorizing `identity:get_user(user_id=b3234aad965e427281c9352b840a0824)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:18.497687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2885fde-976a-4bf2-b639-a3a964da28ce None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:18.499747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 82/161] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:18 2022] GET /identity/v3/users/b3234aad965e427281c9352b840a0824 => generated 247 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:18.505971 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2885fde-976a-4bf2-b639-a3a964da28ce None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:18.537015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:18.628366 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2885fde-976a-4bf2-b639-a3a964da28ce None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:18.632550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e335905-465b-46b5-ad95-672d5678d638 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:18.670929 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034af601-f7c1-47e5-8b29-318c28a5816d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:18.672789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034af601-f7c1-47e5-8b29-318c28a5816d None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:18.673369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034af601-f7c1-47e5-8b29-318c28a5816d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:18.674732 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034af601-f7c1-47e5-8b29-318c28a5816d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:18.698880 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00328824-eee0-4613-b2f1-b514fab9f4da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:18.699782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00328824-eee0-4613-b2f1-b514fab9f4da None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:18.700256 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00328824-eee0-4613-b2f1-b514fab9f4da None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:18.700692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:18.700815 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00328824-eee0-4613-b2f1-b514fab9f4da None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:18.702506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00328824-eee0-4613-b2f1-b514fab9f4da None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:18.704593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00328824-eee0-4613-b2f1-b514fab9f4da None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:18.706600 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 80/162] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:18 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:18.712574 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00328824-eee0-4613-b2f1-b514fab9f4da None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:18.811397 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034af601-f7c1-47e5-8b29-318c28a5816d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:18.812903 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034af601-f7c1-47e5-8b29-318c28a5816d None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:18.922320 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00328824-eee0-4613-b2f1-b514fab9f4da None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:18.947200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:18.989033 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da453b7b-bf1a-4319-a318-16576ff61166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:18.989762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da453b7b-bf1a-4319-a318-16576ff61166 None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:18.990046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da453b7b-bf1a-4319-a318-16576ff61166 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:18.990319 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da453b7b-bf1a-4319-a318-16576ff61166 None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4/users/a1d83d21fa9c44bd83e478aefd3c492a/roles/b5eaad32927541e2a79d4904b5230343` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:18.994431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da453b7b-bf1a-4319-a318-16576ff61166 None admin] RBAC: Authorizing `identity:create_grant(project_id=63ed5323666040089b15189fade45bf4, user_id=a1d83d21fa9c44bd83e478aefd3c492a, role_id=b5eaad32927541e2a79d4904b5230343)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:18.995743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da453b7b-bf1a-4319-a318-16576ff61166 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:19.012711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 83/163] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 507 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:19.014166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd9f808-75af-4cb8-9cf4-ad702575734d None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:19.014551 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd9f808-75af-4cb8-9cf4-ad702575734d None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:19.014836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd9f808-75af-4cb8-9cf4-ad702575734d None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:19.019181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 84/164] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:19.021124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd9f808-75af-4cb8-9cf4-ad702575734d None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:19.064452 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 81/165] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:04:18 2022] PUT /identity/v3/projects/63ed5323666040089b15189fade45bf4/users/a1d83d21fa9c44bd83e478aefd3c492a/roles/b5eaad32927541e2a79d4904b5230343 => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:04:19.065941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bcb72f-9c2c-4d5f-ba1d-49857cf6452c None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:19.066232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bcb72f-9c2c-4d5f-ba1d-49857cf6452c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:19.066509 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bcb72f-9c2c-4d5f-ba1d-49857cf6452c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:19.103042 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-23bcb72f-9c2c-4d5f-ba1d-49857cf6452c None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:19.104239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:19.106052 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-23bcb72f-9c2c-4d5f-ba1d-49857cf6452c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:19.191687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd9f808-75af-4cb8-9cf4-ad702575734d None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:19.236291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d16f63-9c99-478f-a3b5-01cf5c94f92e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:19.237704 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d16f63-9c99-478f-a3b5-01cf5c94f92e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:19.238160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d16f63-9c99-478f-a3b5-01cf5c94f92e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:19.238871 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d16f63-9c99-478f-a3b5-01cf5c94f92e None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:19.240620 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d16f63-9c99-478f-a3b5-01cf5c94f92e None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:19.242926 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d16f63-9c99-478f-a3b5-01cf5c94f92e None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:19.246176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 85/166] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:19 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:19.247482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d16f63-9c99-478f-a3b5-01cf5c94f92e None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:19.255464 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 82/167] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:19.255464 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bcb72f-9c2c-4d5f-ba1d-49857cf6452c None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:19.445069 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d16f63-9c99-478f-a3b5-01cf5c94f92e None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:19.468213 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bcb72f-9c2c-4d5f-ba1d-49857cf6452c None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:19.468613 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:19.471007 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:19.499276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c24622-47cb-4133-b3bc-81d81a33a21d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:19.499983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c24622-47cb-4133-b3bc-81d81a33a21d None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:19.500286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c24622-47cb-4133-b3bc-81d81a33a21d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:19.500763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c24622-47cb-4133-b3bc-81d81a33a21d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:19.519501 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c24622-47cb-4133-b3bc-81d81a33a21d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:19.520876 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac46bc5-fb96-43b8-817d-7cda2447c424 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:19.521509 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac46bc5-fb96-43b8-817d-7cda2447c424 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:19.521787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac46bc5-fb96-43b8-817d-7cda2447c424 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:19.522056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac46bc5-fb96-43b8-817d-7cda2447c424 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:19.522791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac46bc5-fb96-43b8-817d-7cda2447c424 None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=63ed5323666040089b15189fade45bf4, user.id=b3234aad965e427281c9352b840a0824) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:19.523100 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac46bc5-fb96-43b8-817d-7cda2447c424 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:19.525422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-24c24622-47cb-4133-b3bc-81d81a33a21d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:19.525422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:19.525422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:19.525422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:19.525422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:19.525422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:19.525422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:19.525422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:19.525422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:19.525422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:19.525422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:19.525422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:19.525422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:19.525422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:19.525422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:19.525422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:19.526519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:19.526519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:19.526519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:19.526519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:19.526519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:19.526519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:19.526519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:19.526519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:19.526519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:19.526519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:19.526519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:19.526519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:19.526519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:19.526519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:19.526519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:19.526519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:19.527339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:19.527339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:19.527339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:19.527339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:19.527339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:19.527339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:19.527339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:19.527339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:19.527339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:19.527339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:19.527339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:19.527339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:19.527968 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac46bc5-fb96-43b8-817d-7cda2447c424 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:19.528037 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 86/168] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:04:19 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:19.534774 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c24622-47cb-4133-b3bc-81d81a33a21d None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:19.546725 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 83/169] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:19 2022] GET /identity/v3/role_assignments?user.id=b3234aad965e427281c9352b840a0824&scope.project.id=63ed5323666040089b15189fade45bf4&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 608 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:19.664027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c24622-47cb-4133-b3bc-81d81a33a21d None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:19.703107 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6554d4-ca91-4957-8461-42acaf494030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:19.703910 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6554d4-ca91-4957-8461-42acaf494030 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:19.704074 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6554d4-ca91-4957-8461-42acaf494030 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:19.704374 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6554d4-ca91-4957-8461-42acaf494030 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:19.705046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6554d4-ca91-4957-8461-42acaf494030 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:19.705348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6554d4-ca91-4957-8461-42acaf494030 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:19.706750 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6554d4-ca91-4957-8461-42acaf494030 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:19.716944 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 87/170] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:04:19 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:19.722721 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac46bc5-fb96-43b8-817d-7cda2447c424 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:19.741466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6554d4-ca91-4957-8461-42acaf494030 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:19.816511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:19.882670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6554d4-ca91-4957-8461-42acaf494030 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:19.921450 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19f7c24-0818-469f-8216-ae96b033c072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:19.922100 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19f7c24-0818-469f-8216-ae96b033c072 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:19.922381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19f7c24-0818-469f-8216-ae96b033c072 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:19.922657 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19f7c24-0818-469f-8216-ae96b033c072 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:19.923165 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac46bc5-fb96-43b8-817d-7cda2447c424 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:19.948419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19f7c24-0818-469f-8216-ae96b033c072 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:19.954701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-c19f7c24-0818-469f-8216-ae96b033c072 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:19.954701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:19.954701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:19.954701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:19.954701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:19.954701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:19.954701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:19.954701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:19.954701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:19.954701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:19.954701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:19.954701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:19.954701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:19.954701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:19.954701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:19.954701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:19.955814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:19.955814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:19.955814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:19.955814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:19.955814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:19.955814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:19.955814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:19.955814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:19.955814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:19.955814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:19.955814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:19.955814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:19.955814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:19.955814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:19.955814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:19.955814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:19.956849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:19.956849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:19.956849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:19.956849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:19.956849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:19.956849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:19.956849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:19.956849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:19.956849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:19.956849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:19.956849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:19.956849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:19.959471 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 88/171] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:19.963713 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eceaab-6a08-4a15-a763-cbd5fb530f5e None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:19.964528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eceaab-6a08-4a15-a763-cbd5fb530f5e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:19.964948 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eceaab-6a08-4a15-a763-cbd5fb530f5e None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:19.966788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 89/172] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:19 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:19.968179 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88eceaab-6a08-4a15-a763-cbd5fb530f5e None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:19.977794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891b5cc7-0664-48f7-9b67-482244e30759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:19.978758 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b5cc7-0664-48f7-9b67-482244e30759 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:19.978897 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b5cc7-0664-48f7-9b67-482244e30759 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:19.979117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b5cc7-0664-48f7-9b67-482244e30759 None admin] PATH_INFO: `/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:19.984949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891b5cc7-0664-48f7-9b67-482244e30759 None admin] RBAC: Authorizing `identity:get_user(user_id=4e36c8bcb5234cd29a8d01bb6f6d6d38)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:19.986418 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891b5cc7-0664-48f7-9b67-482244e30759 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:19.988166 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 84/173] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:19 2022] GET /identity/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38 => generated 275 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:19.989290 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ec6333-4303-43a4-aebe-d7d6aa354d93 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:19.989566 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ec6333-4303-43a4-aebe-d7d6aa354d93 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:19.989834 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ec6333-4303-43a4-aebe-d7d6aa354d93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:20.025823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-04ec6333-4303-43a4-aebe-d7d6aa354d93 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:20.027040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-04ec6333-4303-43a4-aebe-d7d6aa354d93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:20.061153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:20.091309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:20.093638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88eceaab-6a08-4a15-a763-cbd5fb530f5e None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:20.119585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 85/174] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:20.120959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ec6333-4303-43a4-aebe-d7d6aa354d93 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:20.142278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1350505-c3ef-4efb-8b50-b9c4729f7463 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:20.145062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1350505-c3ef-4efb-8b50-b9c4729f7463 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:20.145338 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1350505-c3ef-4efb-8b50-b9c4729f7463 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:20.145611 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1350505-c3ef-4efb-8b50-b9c4729f7463 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:20.146304 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1350505-c3ef-4efb-8b50-b9c4729f7463 None admin] RBAC: Adding query filter params (name=member) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:20.146602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1350505-c3ef-4efb-8b50-b9c4729f7463 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:20.148244 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1350505-c3ef-4efb-8b50-b9c4729f7463 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:20.161211 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 90/175] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:20.161395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1016ac9a-2678-4ecf-969a-ad28c2cf33cd None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:20.161688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1016ac9a-2678-4ecf-969a-ad28c2cf33cd None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:20.161874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1016ac9a-2678-4ecf-969a-ad28c2cf33cd None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:20.165274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 91/176] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:20.166811 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1016ac9a-2678-4ecf-969a-ad28c2cf33cd None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:20.258887 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ec6333-4303-43a4-aebe-d7d6aa354d93 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:20.302026 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da88364-e9ee-4787-86e6-d752b6b72602 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:20.303015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da88364-e9ee-4787-86e6-d752b6b72602 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:20.303499 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da88364-e9ee-4787-86e6-d752b6b72602 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:20.303996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da88364-e9ee-4787-86e6-d752b6b72602 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:20.352474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:20.393472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da88364-e9ee-4787-86e6-d752b6b72602 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:20.395288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da88364-e9ee-4787-86e6-d752b6b72602 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:20.423123 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1016ac9a-2678-4ecf-969a-ad28c2cf33cd None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:20.467962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c778584c-53a7-4522-8ea4-e2c72d8d165f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:20.468671 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c778584c-53a7-4522-8ea4-e2c72d8d165f None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:20.468943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c778584c-53a7-4522-8ea4-e2c72d8d165f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:20.469209 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c778584c-53a7-4522-8ea4-e2c72d8d165f None admin] PATH_INFO: `/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:20.470331 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c778584c-53a7-4522-8ea4-e2c72d8d165f None admin] RBAC: Authorizing `identity:get_user(user_id=4e36c8bcb5234cd29a8d01bb6f6d6d38)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:20.471765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c778584c-53a7-4522-8ea4-e2c72d8d165f None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:20.473979 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 92/177] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:20 2022] GET /identity/v3/users/4e36c8bcb5234cd29a8d01bb6f6d6d38 => generated 275 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:20.475047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c77e38-810c-41f0-b48f-d8f8d0575c0e None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:20.475327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c77e38-810c-41f0-b48f-d8f8d0575c0e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:20.475591 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c77e38-810c-41f0-b48f-d8f8d0575c0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:20.495692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-14c77e38-810c-41f0-b48f-d8f8d0575c0e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:20.496975 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-14c77e38-810c-41f0-b48f-d8f8d0575c0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:20.515757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:20.526329 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 86/178] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:20.527822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da88364-e9ee-4787-86e6-d752b6b72602 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:20.581052 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 93/179] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:20.582532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c77e38-810c-41f0-b48f-d8f8d0575c0e None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:20.599216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:20.636389 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da88364-e9ee-4787-86e6-d752b6b72602 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:20.673142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36716df-4046-405b-881f-b48b834ea0d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:20.673858 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36716df-4046-405b-881f-b48b834ea0d3 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:20.674425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36716df-4046-405b-881f-b48b834ea0d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:20.674425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36716df-4046-405b-881f-b48b834ea0d3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:20.685352 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36716df-4046-405b-881f-b48b834ea0d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:20.690642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-a36716df-4046-405b-881f-b48b834ea0d3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:20.690642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:20.690642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:20.690642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:20.690642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:20.690642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:20.690642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:20.690642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:20.690642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:20.690642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:20.690642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:20.690642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:20.690642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:20.690642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:20.690642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:20.690642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:20.691552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:20.691552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:20.691552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:20.691552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:20.691552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:20.691552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:20.691552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:20.691552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:20.691552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:20.691552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:20.691552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:20.691552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:20.691552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:20.691552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:20.691552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:20.691552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:20.692408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:20.692408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:20.692408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:20.692408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:20.692408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:20.692408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:20.692408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:20.692408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:20.692408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:20.692408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:20.692408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:20.692408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:20.692408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 87/180] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:04:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:20.693072 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36716df-4046-405b-881f-b48b834ea0d3 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:20.704603 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c77e38-810c-41f0-b48f-d8f8d0575c0e None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:20.745790 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b741d542-c8bc-4278-b8c3-10524021bf0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:20.747105 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b741d542-c8bc-4278-b8c3-10524021bf0e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:20.747631 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b741d542-c8bc-4278-b8c3-10524021bf0e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:20.748136 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b741d542-c8bc-4278-b8c3-10524021bf0e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:20.778052 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:20.830472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36716df-4046-405b-881f-b48b834ea0d3 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:20.833718 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b741d542-c8bc-4278-b8c3-10524021bf0e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:20.836740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b741d542-c8bc-4278-b8c3-10524021bf0e None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:20.892591 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:20.905350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c912fc9-8f57-4090-884e-e72a6e1b2527 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:20.905994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c912fc9-8f57-4090-884e-e72a6e1b2527 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:20.906442 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c912fc9-8f57-4090-884e-e72a6e1b2527 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:20.906785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c912fc9-8f57-4090-884e-e72a6e1b2527 None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:20.908702 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c912fc9-8f57-4090-884e-e72a6e1b2527 None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:20.910732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c912fc9-8f57-4090-884e-e72a6e1b2527 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:20.913288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 88/181] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:20 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:20.914149 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c912fc9-8f57-4090-884e-e72a6e1b2527 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:20.961250 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 94/182] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:20.963069 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1522c862-ce07-4205-ba04-96d5915e210b None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:20.963415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1522c862-ce07-4205-ba04-96d5915e210b None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:20.963902 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1522c862-ce07-4205-ba04-96d5915e210b None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:20.969498 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 95/183] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:20.970585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1522c862-ce07-4205-ba04-96d5915e210b None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:21.034210 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c912fc9-8f57-4090-884e-e72a6e1b2527 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:21.034791 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:21.071174 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4ac953-e627-4afc-9250-b3b9db596eec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:21.072698 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4ac953-e627-4afc-9250-b3b9db596eec None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:21.072975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4ac953-e627-4afc-9250-b3b9db596eec None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:21.073250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4ac953-e627-4afc-9250-b3b9db596eec None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:21.073920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4ac953-e627-4afc-9250-b3b9db596eec None admin] RBAC: Adding query filter params (name=admin) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:21.074222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4ac953-e627-4afc-9250-b3b9db596eec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:21.075652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4ac953-e627-4afc-9250-b3b9db596eec None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:21.084304 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 89/184] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:21.085129 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4ac953-e627-4afc-9250-b3b9db596eec None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:21.091081 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1522c862-ce07-4205-ba04-96d5915e210b None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:21.128426 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3de286-cbd1-4451-b5a4-d60d078910b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:21.129047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3de286-cbd1-4451-b5a4-d60d078910b4 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:21.129325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3de286-cbd1-4451-b5a4-d60d078910b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:21.129594 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3de286-cbd1-4451-b5a4-d60d078910b4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:21.130324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3de286-cbd1-4451-b5a4-d60d078910b4 None admin] RBAC: Adding query filter params (role.id=fbc5961470b74a85bde6be2171f5f08f, scope.project.id=63ed5323666040089b15189fade45bf4, user.id=4e36c8bcb5234cd29a8d01bb6f6d6d38) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:21.130629 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3de286-cbd1-4451-b5a4-d60d078910b4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:21.132067 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3de286-cbd1-4451-b5a4-d60d078910b4 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:21.139266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 96/185] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:20 2022] GET /identity/v3/role_assignments?user.id=4e36c8bcb5234cd29a8d01bb6f6d6d38&scope.project.id=63ed5323666040089b15189fade45bf4&role.id=fbc5961470b74a85bde6be2171f5f08f => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:21.150317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3884510a-1f05-44b1-85b6-deb0e5786501 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:21.150646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3884510a-1f05-44b1-85b6-deb0e5786501 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:21.150863 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3884510a-1f05-44b1-85b6-deb0e5786501 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:21.155367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:21.173548 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-3884510a-1f05-44b1-85b6-deb0e5786501 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:21.175171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3884510a-1f05-44b1-85b6-deb0e5786501 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:21.218281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4ac953-e627-4afc-9250-b3b9db596eec None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:21.272589 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b119f1-85cc-4858-9ea3-7737dc1f3a47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:21.273515 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b119f1-85cc-4858-9ea3-7737dc1f3a47 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:21.273794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b119f1-85cc-4858-9ea3-7737dc1f3a47 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:21.274069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b119f1-85cc-4858-9ea3-7737dc1f3a47 None admin] PATH_INFO: `/v3/projects/407a13bf0fbc456794954f352ca2577a` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:21.275997 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b119f1-85cc-4858-9ea3-7737dc1f3a47 None admin] RBAC: Authorizing `identity:get_project(project_id=407a13bf0fbc456794954f352ca2577a)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:21.277485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b119f1-85cc-4858-9ea3-7737dc1f3a47 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:21.279457 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 90/186] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:21 2022] GET /identity/v3/projects/407a13bf0fbc456794954f352ca2577a => generated 312 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:21.279997 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b119f1-85cc-4858-9ea3-7737dc1f3a47 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:21.294386 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 97/187] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:21.294386 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3884510a-1f05-44b1-85b6-deb0e5786501 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:21.371486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:21.434498 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3884510a-1f05-44b1-85b6-deb0e5786501 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:21.440283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b119f1-85cc-4858-9ea3-7737dc1f3a47 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:21.479109 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14e17d6-176d-4286-b458-bf7d86d829b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:21.480175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14e17d6-176d-4286-b458-bf7d86d829b6 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:21.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14e17d6-176d-4286-b458-bf7d86d829b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:21.481110 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14e17d6-176d-4286-b458-bf7d86d829b6 None admin] PATH_INFO: `/v3/users/1a140a19a341404ca775cbacdb00abf7` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:21.483743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:21.488920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7113b6ce-4dc7-42af-8786-14fc4f47611e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:21.492161 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7113b6ce-4dc7-42af-8786-14fc4f47611e None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:21.492161 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7113b6ce-4dc7-42af-8786-14fc4f47611e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:21.492161 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7113b6ce-4dc7-42af-8786-14fc4f47611e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:21.499650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14e17d6-176d-4286-b458-bf7d86d829b6 None admin] RBAC: Authorizing `identity:get_user(user_id=1a140a19a341404ca775cbacdb00abf7)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:21.501380 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14e17d6-176d-4286-b458-bf7d86d829b6 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:21.503291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 98/188] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:21 2022] GET /identity/v3/users/1a140a19a341404ca775cbacdb00abf7 => generated 283 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:21.504231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14e17d6-176d-4286-b458-bf7d86d829b6 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:21.504406 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7113b6ce-4dc7-42af-8786-14fc4f47611e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:21.511288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-7113b6ce-4dc7-42af-8786-14fc4f47611e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:21.511288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:21.511288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:21.511288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:21.511288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:21.511288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:21.511288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:21.511288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:21.511288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:21.511288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:21.511288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:21.511288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:21.511288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:21.511288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:21.511288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:21.511288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:21.512502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:21.512502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:21.512502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:21.512502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:21.512502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:21.512502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:21.512502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:21.512502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:21.512502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:21.512502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:21.512502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:21.512502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:21.512502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:21.512502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:21.512502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:21.512502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:21.513503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:21.513503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:21.513503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:21.513503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:21.513503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:21.513503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:21.513503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:21.513503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:21.513503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:21.513503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:21.513503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:21.513503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:21.513503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 91/189] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:04:21 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:21.513503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7113b6ce-4dc7-42af-8786-14fc4f47611e None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:21.648770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7113b6ce-4dc7-42af-8786-14fc4f47611e None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:21.657680 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14e17d6-176d-4286-b458-bf7d86d829b6 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:21.669339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:21.700868 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7d579c-5f45-4599-a8c5-257725d7bf71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:21.701483 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c56545-d074-4fdc-a730-a9a743e98b33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:21.701624 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7d579c-5f45-4599-a8c5-257725d7bf71 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:21.702058 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7d579c-5f45-4599-a8c5-257725d7bf71 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:21.702150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c56545-d074-4fdc-a730-a9a743e98b33 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:21.702491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7d579c-5f45-4599-a8c5-257725d7bf71 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:21.702580 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c56545-d074-4fdc-a730-a9a743e98b33 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:21.702827 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c56545-d074-4fdc-a730-a9a743e98b33 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:21.703643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c56545-d074-4fdc-a730-a9a743e98b33 None admin] RBAC: Adding query filter params (role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe, scope.project.id=407a13bf0fbc456794954f352ca2577a, user.id=4e36c8bcb5234cd29a8d01bb6f6d6d38) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:21.704020 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c56545-d074-4fdc-a730-a9a743e98b33 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:21.705571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c56545-d074-4fdc-a730-a9a743e98b33 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:21.720154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 99/190] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:21 2022] GET /identity/v3/role_assignments?user.id=4e36c8bcb5234cd29a8d01bb6f6d6d38&scope.project.id=407a13bf0fbc456794954f352ca2577a&role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 608 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:21.720154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c56545-d074-4fdc-a730-a9a743e98b33 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:21.784541 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:21.806721 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7d579c-5f45-4599-a8c5-257725d7bf71 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:21.808351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7d579c-5f45-4599-a8c5-257725d7bf71 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:21.843163 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c56545-d074-4fdc-a730-a9a743e98b33 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:21.886944 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9b618a-a2dd-4a0b-b1d7-276ea993af44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:21.887581 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9b618a-a2dd-4a0b-b1d7-276ea993af44 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:21.887874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9b618a-a2dd-4a0b-b1d7-276ea993af44 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:21.888683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9b618a-a2dd-4a0b-b1d7-276ea993af44 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:21.889074 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9b618a-a2dd-4a0b-b1d7-276ea993af44 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:21.889377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9b618a-a2dd-4a0b-b1d7-276ea993af44 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:21.890796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9b618a-a2dd-4a0b-b1d7-276ea993af44 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:21.902373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 100/191] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:04:21 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:21.902823 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9b618a-a2dd-4a0b-b1d7-276ea993af44 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:21.944904 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:21.948545 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 92/192] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:21.953789 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a96661-c520-498b-bb5f-f86703ee03b4 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:21.954076 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a96661-c520-498b-bb5f-f86703ee03b4 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:21.954355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a96661-c520-498b-bb5f-f86703ee03b4 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:21.955883 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 93/193] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:21.957127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a96661-c520-498b-bb5f-f86703ee03b4 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:22.017400 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9b618a-a2dd-4a0b-b1d7-276ea993af44 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:22.049337 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:22.053076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608e6dc1-95c3-4fe2-9ee7-7132f7dd6a46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:22.053715 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608e6dc1-95c3-4fe2-9ee7-7132f7dd6a46 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:22.053990 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608e6dc1-95c3-4fe2-9ee7-7132f7dd6a46 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:22.054261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608e6dc1-95c3-4fe2-9ee7-7132f7dd6a46 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:22.058589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608e6dc1-95c3-4fe2-9ee7-7132f7dd6a46 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:22.058589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608e6dc1-95c3-4fe2-9ee7-7132f7dd6a46 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:22.059152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 101/194] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:04:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:22.059754 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608e6dc1-95c3-4fe2-9ee7-7132f7dd6a46 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:22.075039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a96661-c520-498b-bb5f-f86703ee03b4 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:22.116354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16543c82-5e5f-4c58-8ea6-d5e487122f5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:22.116988 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16543c82-5e5f-4c58-8ea6-d5e487122f5c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:22.119158 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16543c82-5e5f-4c58-8ea6-d5e487122f5c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:22.119439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16543c82-5e5f-4c58-8ea6-d5e487122f5c None admin] PATH_INFO: `/v3/users/1a140a19a341404ca775cbacdb00abf7` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:22.120695 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16543c82-5e5f-4c58-8ea6-d5e487122f5c None admin] RBAC: Authorizing `identity:get_user(user_id=1a140a19a341404ca775cbacdb00abf7)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:22.122205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16543c82-5e5f-4c58-8ea6-d5e487122f5c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:22.124017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 94/195] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:21 2022] GET /identity/v3/users/1a140a19a341404ca775cbacdb00abf7 => generated 283 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:22.125112 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27620bc0-c3f6-4d9a-a71c-8c1c489b20ef None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:22.125387 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27620bc0-c3f6-4d9a-a71c-8c1c489b20ef None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:22.125651 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27620bc0-c3f6-4d9a-a71c-8c1c489b20ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:22.144727 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-27620bc0-c3f6-4d9a-a71c-8c1c489b20ef None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:22.145957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-27620bc0-c3f6-4d9a-a71c-8c1c489b20ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:22.175426 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:22.178347 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608e6dc1-95c3-4fe2-9ee7-7132f7dd6a46 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:22.216759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376fd2fb-27ae-409b-8ce5-16f5c9ad61db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:22.217391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376fd2fb-27ae-409b-8ce5-16f5c9ad61db None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:22.217663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376fd2fb-27ae-409b-8ce5-16f5c9ad61db None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:22.217933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376fd2fb-27ae-409b-8ce5-16f5c9ad61db None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:22.219024 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 95/196] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:22.219381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376fd2fb-27ae-409b-8ce5-16f5c9ad61db None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:22.219841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27620bc0-c3f6-4d9a-a71c-8c1c489b20ef None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:22.223350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376fd2fb-27ae-409b-8ce5-16f5c9ad61db None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:22.225121 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 102/197] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:22 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:22.225910 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376fd2fb-27ae-409b-8ce5-16f5c9ad61db None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:22.316706 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:22.354714 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27620bc0-c3f6-4d9a-a71c-8c1c489b20ef None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:22.361962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376fd2fb-27ae-409b-8ce5-16f5c9ad61db None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:22.404352 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9625eb5-ac92-4791-8d26-6f5374b14e46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:22.405384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9625eb5-ac92-4791-8d26-6f5374b14e46 None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:22.405877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9625eb5-ac92-4791-8d26-6f5374b14e46 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:22.406356 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9625eb5-ac92-4791-8d26-6f5374b14e46 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:22.407403 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9625eb5-ac92-4791-8d26-6f5374b14e46 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:22.409842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9625eb5-ac92-4791-8d26-6f5374b14e46 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:22.416149 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f813d63-e8c9-4282-a0fe-d36578996e2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:22.416779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f813d63-e8c9-4282-a0fe-d36578996e2d None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:22.417082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f813d63-e8c9-4282-a0fe-d36578996e2d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:22.417350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f813d63-e8c9-4282-a0fe-d36578996e2d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:22.456583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:22.490953 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f813d63-e8c9-4282-a0fe-d36578996e2d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:22.492464 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f813d63-e8c9-4282-a0fe-d36578996e2d None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:22.506833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 96/198] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:04:22 2022] POST /identity/v3/users => generated 297 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:04:22.507667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9625eb5-ac92-4791-8d26-6f5374b14e46 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:22.625125 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 103/199] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:22.629982 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9625eb5-ac92-4791-8d26-6f5374b14e46 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:22.630831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f813d63-e8c9-4282-a0fe-d36578996e2d None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:22.673537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d6b200-8710-44cd-9888-cd32b2eb216b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:22.674547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d6b200-8710-44cd-9888-cd32b2eb216b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:22.674547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d6b200-8710-44cd-9888-cd32b2eb216b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:22.678453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d6b200-8710-44cd-9888-cd32b2eb216b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:22.678453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d6b200-8710-44cd-9888-cd32b2eb216b None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=ea5d616ff88c481baff44c2b3840d3d4, user.id=1a140a19a341404ca775cbacdb00abf7) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:22.678453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d6b200-8710-44cd-9888-cd32b2eb216b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:22.678759 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d6b200-8710-44cd-9888-cd32b2eb216b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:22.685152 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 97/200] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:22 2022] GET /identity/v3/role_assignments?user.id=1a140a19a341404ca775cbacdb00abf7&scope.project.id=ea5d616ff88c481baff44c2b3840d3d4&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:22.686112 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d6b200-8710-44cd-9888-cd32b2eb216b None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:22.689284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:22.719459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:22.765105 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f813d63-e8c9-4282-a0fe-d36578996e2d None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:22.821074 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8d198f-076f-4f99-8748-bad314b4dcf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:22.822334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8d198f-076f-4f99-8748-bad314b4dcf1 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:22.822611 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8d198f-076f-4f99-8748-bad314b4dcf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:22.822881 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8d198f-076f-4f99-8748-bad314b4dcf1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:22.837687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8d198f-076f-4f99-8748-bad314b4dcf1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:22.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-9b8d198f-076f-4f99-8748-bad314b4dcf1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:22.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:22.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:22.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:22.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:22.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:22.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:22.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:22.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:22.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:22.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:22.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:22.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:22.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:22.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:22.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:22.843793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:22.843793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:22.843793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:22.843793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:22.843793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:22.843793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:22.843793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:22.843793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:22.843793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:22.843793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:22.843793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:22.843793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:22.843793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:22.843793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:22.843793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:22.843793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:22.845415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:22.845415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:22.845415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:22.845415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:22.845415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:22.845415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:22.845415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:22.845415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:22.845415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:22.845415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:22.845415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:22.845415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:22.845415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 104/201] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:22 2022] GET /identity/v3/roles/reader => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:22.848315 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d6b200-8710-44cd-9888-cd32b2eb216b None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:22.852883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8d198f-076f-4f99-8748-bad314b4dcf1 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:22.899893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f00f81e-37bc-4f02-93a4-c14e5f7f12ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:22.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f00f81e-37bc-4f02-93a4-c14e5f7f12ed None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:22.900973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f00f81e-37bc-4f02-93a4-c14e5f7f12ed None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:22.901244 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f00f81e-37bc-4f02-93a4-c14e5f7f12ed None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:22.902414 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f00f81e-37bc-4f02-93a4-c14e5f7f12ed None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:22.903935 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f00f81e-37bc-4f02-93a4-c14e5f7f12ed None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:22.905990 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 98/202] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:22 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:22.913353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f00f81e-37bc-4f02-93a4-c14e5f7f12ed None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:22.990963 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8d198f-076f-4f99-8748-bad314b4dcf1 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:23.008364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:23.022951 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:23.031318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b6efd9-839d-439f-a553-12bd2038b2ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:23.032277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b6efd9-839d-439f-a553-12bd2038b2ad None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:23.032765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b6efd9-839d-439f-a553-12bd2038b2ad None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:23.033325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b6efd9-839d-439f-a553-12bd2038b2ad None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:23.034195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b6efd9-839d-439f-a553-12bd2038b2ad None admin] RBAC: Adding query filter params (name=reader) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:23.034719 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b6efd9-839d-439f-a553-12bd2038b2ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:23.036341 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b6efd9-839d-439f-a553-12bd2038b2ad None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:23.045390 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f00f81e-37bc-4f02-93a4-c14e5f7f12ed None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:23.047019 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 105/203] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:22 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:23.052755 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b6efd9-839d-439f-a553-12bd2038b2ad None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:23.086653 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b18df2-e085-4695-a6fa-d12bae54c823 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:23.087308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b18df2-e085-4695-a6fa-d12bae54c823 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:23.087660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b18df2-e085-4695-a6fa-d12bae54c823 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:23.087923 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b18df2-e085-4695-a6fa-d12bae54c823 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:23.088689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b18df2-e085-4695-a6fa-d12bae54c823 None admin] RBAC: Adding query filter params (role.id=fbc5961470b74a85bde6be2171f5f08f, scope.project.id=ea5d616ff88c481baff44c2b3840d3d4, user.id=1a140a19a341404ca775cbacdb00abf7) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:23.089000 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b18df2-e085-4695-a6fa-d12bae54c823 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:23.090666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b18df2-e085-4695-a6fa-d12bae54c823 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:23.096864 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 99/204] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:22 2022] GET /identity/v3/role_assignments?user.id=1a140a19a341404ca775cbacdb00abf7&scope.project.id=ea5d616ff88c481baff44c2b3840d3d4&role.id=fbc5961470b74a85bde6be2171f5f08f => generated 259 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:23.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b6efd9-839d-439f-a553-12bd2038b2ad None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:23.210372 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d58240f-5a94-4fed-b218-d1cb28342596 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:23.211385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d58240f-5a94-4fed-b218-d1cb28342596 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:23.211888 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d58240f-5a94-4fed-b218-d1cb28342596 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:23.212420 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d58240f-5a94-4fed-b218-d1cb28342596 None admin] PATH_INFO: `/v3/users/a1d83d21fa9c44bd83e478aefd3c492a` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d58240f-5a94-4fed-b218-d1cb28342596 None admin] RBAC: Authorizing `identity:get_user(user_id=a1d83d21fa9c44bd83e478aefd3c492a)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:23.215621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d58240f-5a94-4fed-b218-d1cb28342596 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:23.217973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 106/205] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:23 2022] GET /identity/v3/users/a1d83d21fa9c44bd83e478aefd3c492a => generated 289 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:23.225345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b18df2-e085-4695-a6fa-d12bae54c823 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:23.372331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b18df2-e085-4695-a6fa-d12bae54c823 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:23.420807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b970337a-e495-4d93-90b4-3915ae283641 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:23.421791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b970337a-e495-4d93-90b4-3915ae283641 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:23.422287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b970337a-e495-4d93-90b4-3915ae283641 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:23.422757 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b970337a-e495-4d93-90b4-3915ae283641 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:23.430461 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:23.528761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b970337a-e495-4d93-90b4-3915ae283641 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:23.530210 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b970337a-e495-4d93-90b4-3915ae283641 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:23.650952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 100/206] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:23.656500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d58240f-5a94-4fed-b218-d1cb28342596 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:23.753893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:23.800153 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d58240f-5a94-4fed-b218-d1cb28342596 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:23.836581 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e7b4fa-e03d-4ece-98c9-1ddbc3392ac9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:23.837242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e7b4fa-e03d-4ece-98c9-1ddbc3392ac9 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:23.837519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e7b4fa-e03d-4ece-98c9-1ddbc3392ac9 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:23.837793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e7b4fa-e03d-4ece-98c9-1ddbc3392ac9 None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:23.838933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e7b4fa-e03d-4ece-98c9-1ddbc3392ac9 None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:23.840397 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e7b4fa-e03d-4ece-98c9-1ddbc3392ac9 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:23.842395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 107/207] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:23 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:23.849238 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b970337a-e495-4d93-90b4-3915ae283641 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:23.916051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:23.988425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b970337a-e495-4d93-90b4-3915ae283641 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:24.034237 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6ab94b-931e-452f-8f41-5f77e1be75a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:24.035629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ab94b-931e-452f-8f41-5f77e1be75a4 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:24.036131 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ab94b-931e-452f-8f41-5f77e1be75a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:24.036597 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ab94b-931e-452f-8f41-5f77e1be75a4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:24.037549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6ab94b-931e-452f-8f41-5f77e1be75a4 None admin] RBAC: Adding query filter params (role.id=b5eaad32927541e2a79d4904b5230343, scope.project.id=63ed5323666040089b15189fade45bf4, user.id=a1d83d21fa9c44bd83e478aefd3c492a) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:24.038047 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6ab94b-931e-452f-8f41-5f77e1be75a4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:24.039682 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6ab94b-931e-452f-8f41-5f77e1be75a4 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:24.047279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 101/208] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:23 2022] GET /identity/v3/role_assignments?user.id=a1d83d21fa9c44bd83e478aefd3c492a&scope.project.id=63ed5323666040089b15189fade45bf4&role.id=b5eaad32927541e2a79d4904b5230343 => generated 608 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:24.982531 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76771038-6fb2-48fb-99e3-35740ec0b441 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:24.983093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76771038-6fb2-48fb-99e3-35740ec0b441 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:24.983361 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76771038-6fb2-48fb-99e3-35740ec0b441 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:24.985218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 108/209] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:24.990993 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc2f943-3350-419e-811c-012fa581b906 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:24.991284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc2f943-3350-419e-811c-012fa581b906 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:24.991541 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc2f943-3350-419e-811c-012fa581b906 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:24.994843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625cb5ae-52af-46b1-bdb2-df5c77507bdb None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:24.995364 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625cb5ae-52af-46b1-bdb2-df5c77507bdb None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:24.995860 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625cb5ae-52af-46b1-bdb2-df5c77507bdb None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:24.997747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 109/210] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:04:25.003836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bbaa3d-5d84-4375-b631-551eecd98fa8 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:25.004727 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bbaa3d-5d84-4375-b631-551eecd98fa8 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:25.005453 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bbaa3d-5d84-4375-b631-551eecd98fa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:25.012135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-9fc2f943-3350-419e-811c-012fa581b906 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:25.013321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9fc2f943-3350-419e-811c-012fa581b906 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:25.026577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-82bbaa3d-5d84-4375-b631-551eecd98fa8 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:25.028160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-82bbaa3d-5d84-4375-b631-551eecd98fa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:25.090329 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 102/211] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:25.106826 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 110/212] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:04:25.141775 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afec301a-6846-4804-aaa3-17df7291b11b None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:25.142034 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afec301a-6846-4804-aaa3-17df7291b11b None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:25.142302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afec301a-6846-4804-aaa3-17df7291b11b None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:25.144153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 103/213] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:25.150391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514f88fe-ee79-4e34-82c2-bbc5a2787143 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:25.151184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514f88fe-ee79-4e34-82c2-bbc5a2787143 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:25.151872 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514f88fe-ee79-4e34-82c2-bbc5a2787143 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:25.173878 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-514f88fe-ee79-4e34-82c2-bbc5a2787143 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:25.176230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-514f88fe-ee79-4e34-82c2-bbc5a2787143 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:25.241309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:25.260981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 111/214] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:25.372439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afec301a-6846-4804-aaa3-17df7291b11b None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:25.409914 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:25.434301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514f88fe-ee79-4e34-82c2-bbc5a2787143 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:25.520491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afec301a-6846-4804-aaa3-17df7291b11b None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:25.564767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514f88fe-ee79-4e34-82c2-bbc5a2787143 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:25.565928 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b582bf-8ec9-4716-ab16-225fe22cb68b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:25.566568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b582bf-8ec9-4716-ab16-225fe22cb68b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:25.566877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b582bf-8ec9-4716-ab16-225fe22cb68b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:25.567144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b582bf-8ec9-4716-ab16-225fe22cb68b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:25.578002 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b582bf-8ec9-4716-ab16-225fe22cb68b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:25.582837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-10b582bf-8ec9-4716-ab16-225fe22cb68b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:25.582837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:25.582837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:25.582837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:25.582837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:25.582837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:25.582837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:25.582837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:25.582837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:25.582837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:25.582837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:25.582837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:25.582837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:25.582837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:25.582837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:25.582837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:25.583769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:25.583769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:25.583769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:25.583769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:25.583769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:25.583769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:25.583769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:25.583769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:25.583769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:25.583769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:25.583769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:25.583769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:25.583769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:25.583769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:25.583769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:25.583769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:25.584896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:25.584896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:25.584896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:25.584896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:25.584896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:25.584896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:25.584896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:25.584896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:25.584896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:25.584896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:25.584896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:25.584896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:25.584896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 104/215] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:04:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:25.585571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b7c60a-7966-4037-a178-5b41c01e5588 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:25.585637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b7c60a-7966-4037-a178-5b41c01e5588 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:25.585994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b7c60a-7966-4037-a178-5b41c01e5588 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:25.587128 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 105/216] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:25.587913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b7c60a-7966-4037-a178-5b41c01e5588 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:25.601110 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d60a441-126a-4006-bb52-bd86f8e8706f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:25.601749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d60a441-126a-4006-bb52-bd86f8e8706f None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:25.602021 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d60a441-126a-4006-bb52-bd86f8e8706f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:25.602288 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d60a441-126a-4006-bb52-bd86f8e8706f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:25.613755 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d60a441-126a-4006-bb52-bd86f8e8706f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:25.619062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-1d60a441-126a-4006-bb52-bd86f8e8706f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:25.619062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:25.619062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:25.619062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:25.619062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:25.619062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:25.619062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:25.619062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:25.619062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:25.619062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:25.619062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:25.619062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:25.619062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:25.619062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:25.619062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:25.619062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:25.620062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:25.620062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:25.620062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:25.620062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:25.620062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:25.620062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:25.620062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:25.620062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:25.620062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:25.620062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:25.620062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:25.620062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:25.620062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:25.620062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:25.620062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:25.620062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:25.621265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:25.621265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:25.621265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:25.621265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:25.621265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:25.621265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:25.621265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:25.621265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:25.621265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:25.621265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:25.621265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:25.621265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:25.621265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 112/217] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:25 2022] GET /identity/v3/roles/member => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:25.622332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d60a441-126a-4006-bb52-bd86f8e8706f None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:25.669916 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:25.697957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:25.711202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b7c60a-7966-4037-a178-5b41c01e5588 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:25.726940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d60a441-126a-4006-bb52-bd86f8e8706f None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:25.746884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df64dba-e20c-4b51-a864-2ac018a553c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:25.747519 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df64dba-e20c-4b51-a864-2ac018a553c1 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:25.747794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df64dba-e20c-4b51-a864-2ac018a553c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:25.748062 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df64dba-e20c-4b51-a864-2ac018a553c1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:25.749209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df64dba-e20c-4b51-a864-2ac018a553c1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:25.750638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df64dba-e20c-4b51-a864-2ac018a553c1 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:25.752247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 106/218] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:04:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:25.757831 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bca50d-06cb-42cf-b221-e6c60c7099c3 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:25.758114 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bca50d-06cb-42cf-b221-e6c60c7099c3 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:25.758383 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bca50d-06cb-42cf-b221-e6c60c7099c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:25.763674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf873e7-2211-426a-b360-a46a67d66eda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:25.764736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf873e7-2211-426a-b360-a46a67d66eda None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:25.764854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf873e7-2211-426a-b360-a46a67d66eda None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:25.765048 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf873e7-2211-426a-b360-a46a67d66eda None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:25.766583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf873e7-2211-426a-b360-a46a67d66eda None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:25.766887 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf873e7-2211-426a-b360-a46a67d66eda None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:25.768314 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf873e7-2211-426a-b360-a46a67d66eda None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:25.776774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-89bca50d-06cb-42cf-b221-e6c60c7099c3 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:25.777664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 113/219] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:25.777913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-89bca50d-06cb-42cf-b221-e6c60c7099c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:25.778490 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf873e7-2211-426a-b360-a46a67d66eda None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:25.852578 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 107/220] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:25.854481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bca50d-06cb-42cf-b221-e6c60c7099c3 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:25.901374 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf873e7-2211-426a-b360-a46a67d66eda None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:25.911899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:25.938823 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6d787b-5294-46c2-aeb2-6c7ebac5d287 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:25.939459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6d787b-5294-46c2-aeb2-6c7ebac5d287 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:25.939730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6d787b-5294-46c2-aeb2-6c7ebac5d287 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:25.940015 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6d787b-5294-46c2-aeb2-6c7ebac5d287 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:25.941065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6d787b-5294-46c2-aeb2-6c7ebac5d287 None admin] RBAC: Adding query filter params (name=member) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:25.941632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6d787b-5294-46c2-aeb2-6c7ebac5d287 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:25.943203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6d787b-5294-46c2-aeb2-6c7ebac5d287 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:25.952695 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 114/221] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:25.953616 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6d787b-5294-46c2-aeb2-6c7ebac5d287 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:25.997457 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:26.012995 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bca50d-06cb-42cf-b221-e6c60c7099c3 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:26.050509 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63f6e78-3d03-455a-acf7-8bac03fe45cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:26.051036 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63f6e78-3d03-455a-acf7-8bac03fe45cb None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:26.051308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63f6e78-3d03-455a-acf7-8bac03fe45cb None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:26.051575 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63f6e78-3d03-455a-acf7-8bac03fe45cb None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:26.052237 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63f6e78-3d03-455a-acf7-8bac03fe45cb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:26.053687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63f6e78-3d03-455a-acf7-8bac03fe45cb None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:26.074542 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6d787b-5294-46c2-aeb2-6c7ebac5d287 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:26.111357 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ae11a3-2285-4eaa-84fe-45c20b0ef436 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:26.111999 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ae11a3-2285-4eaa-84fe-45c20b0ef436 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:26.112300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ae11a3-2285-4eaa-84fe-45c20b0ef436 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:26.112572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ae11a3-2285-4eaa-84fe-45c20b0ef436 None admin] PATH_INFO: `/v3/users/1a140a19a341404ca775cbacdb00abf7` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:26.113779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ae11a3-2285-4eaa-84fe-45c20b0ef436 None admin] RBAC: Authorizing `identity:get_user(user_id=1a140a19a341404ca775cbacdb00abf7)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:26.115216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ae11a3-2285-4eaa-84fe-45c20b0ef436 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:26.117043 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 115/222] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:25 2022] GET /identity/v3/users/1a140a19a341404ca775cbacdb00abf7 => generated 283 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:26.119068 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ae11a3-2285-4eaa-84fe-45c20b0ef436 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:26.158458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 108/223] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:04:25 2022] POST /identity/v3/users => generated 297 bytes in 305 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:04:26.159312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63f6e78-3d03-455a-acf7-8bac03fe45cb None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:26.174608 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:26.266688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ae11a3-2285-4eaa-84fe-45c20b0ef436 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:26.303575 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63f6e78-3d03-455a-acf7-8bac03fe45cb None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:26.318534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5652df61-df14-4d8c-b178-1dc3dd7e72b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:26.322740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5652df61-df14-4d8c-b178-1dc3dd7e72b3 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:26.322740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5652df61-df14-4d8c-b178-1dc3dd7e72b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:26.322740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5652df61-df14-4d8c-b178-1dc3dd7e72b3 None admin] PATH_INFO: `/v3/users/efe39ad3c10143819e66dd0b1e8eb7e6` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:26.335494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5652df61-df14-4d8c-b178-1dc3dd7e72b3 None admin] RBAC: Authorizing `identity:get_user(user_id=efe39ad3c10143819e66dd0b1e8eb7e6)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:26.337546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5652df61-df14-4d8c-b178-1dc3dd7e72b3 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:26.340168 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 116/224] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:26 2022] GET /identity/v3/users/efe39ad3c10143819e66dd0b1e8eb7e6 => generated 297 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:26.341021 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5652df61-df14-4d8c-b178-1dc3dd7e72b3 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:26.410061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d53725b-8bf4-40db-818d-c91b2fce6244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:26.410715 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d53725b-8bf4-40db-818d-c91b2fce6244 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:26.410995 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d53725b-8bf4-40db-818d-c91b2fce6244 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:26.411267 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d53725b-8bf4-40db-818d-c91b2fce6244 None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:26.412531 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d53725b-8bf4-40db-818d-c91b2fce6244 None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:26.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d53725b-8bf4-40db-818d-c91b2fce6244 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:26.415817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 109/225] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:26 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:26.417489 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d53725b-8bf4-40db-818d-c91b2fce6244 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:26.432381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:26.473876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:26.484040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5652df61-df14-4d8c-b178-1dc3dd7e72b3 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:26.530903 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b3d9c0-bcea-4f75-bc7d-a5114aa05e9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:26.531660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b3d9c0-bcea-4f75-bc7d-a5114aa05e9d None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:26.532012 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b3d9c0-bcea-4f75-bc7d-a5114aa05e9d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:26.532317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b3d9c0-bcea-4f75-bc7d-a5114aa05e9d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:26.561734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d53725b-8bf4-40db-818d-c91b2fce6244 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:26.563711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b3d9c0-bcea-4f75-bc7d-a5114aa05e9d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:26.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-16b3d9c0-bcea-4f75-bc7d-a5114aa05e9d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:26.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:26.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:26.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:26.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:26.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:26.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:26.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:26.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:26.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:26.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:26.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:26.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:26.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:26.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:26.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:26.572239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:26.572239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:26.572239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:26.572239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:26.572239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:26.572239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:26.572239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:26.572239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:26.572239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:26.572239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:26.572239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:26.572239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:26.572239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:26.572239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:26.572239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:26.572239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:26.573216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:26.573216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:26.573216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:26.573216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:26.573216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:26.573216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:26.573216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:26.573216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:26.573216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:26.573216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:26.573216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:26.573216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:26.573216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 117/226] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:04:26 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:26.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b3d9c0-bcea-4f75-bc7d-a5114aa05e9d None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:26.604678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1d91dd-c2c3-4807-a3fe-117c890adc84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:26.605410 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1d91dd-c2c3-4807-a3fe-117c890adc84 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:26.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1d91dd-c2c3-4807-a3fe-117c890adc84 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:26.606100 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1d91dd-c2c3-4807-a3fe-117c890adc84 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:26.709800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1d91dd-c2c3-4807-a3fe-117c890adc84 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:26.712185 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1d91dd-c2c3-4807-a3fe-117c890adc84 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:26.726037 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b3d9c0-bcea-4f75-bc7d-a5114aa05e9d None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:26.728550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:26.772011 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc3cc62-754c-4d04-916e-09a3c083b53d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:26.773444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc3cc62-754c-4d04-916e-09a3c083b53d None admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:26.773958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc3cc62-754c-4d04-916e-09a3c083b53d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:26.775144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc3cc62-754c-4d04-916e-09a3c083b53d None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4/users/1a140a19a341404ca775cbacdb00abf7/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:26.777747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc3cc62-754c-4d04-916e-09a3c083b53d None admin] RBAC: Authorizing `identity:create_grant(project_id=ea5d616ff88c481baff44c2b3840d3d4, user_id=1a140a19a341404ca775cbacdb00abf7, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:26.779841 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc3cc62-754c-4d04-916e-09a3c083b53d None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:26.847672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 110/227] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:26.848490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1d91dd-c2c3-4807-a3fe-117c890adc84 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:26.896905 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 118/228] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:04:26 2022] PUT /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4/users/1a140a19a341404ca775cbacdb00abf7/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:04:26.897828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc3cc62-754c-4d04-916e-09a3c083b53d None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:26.927800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:27.002235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1d91dd-c2c3-4807-a3fe-117c890adc84 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:27.029359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:27.061511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc3cc62-754c-4d04-916e-09a3c083b53d None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:27.106697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc930c7-1eac-44d5-a072-97460b9789ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:27.107675 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc930c7-1eac-44d5-a072-97460b9789ad None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:27.108191 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc930c7-1eac-44d5-a072-97460b9789ad None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:27.108662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc930c7-1eac-44d5-a072-97460b9789ad None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:27.109579 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc930c7-1eac-44d5-a072-97460b9789ad None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:27.110133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc930c7-1eac-44d5-a072-97460b9789ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:27.111753 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc930c7-1eac-44d5-a072-97460b9789ad None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:27.121619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 111/229] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:04:26 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:27.128448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc930c7-1eac-44d5-a072-97460b9789ad None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:27.129058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87d79f7-b4d6-49b3-a117-58e977328dd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:27.129719 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87d79f7-b4d6-49b3-a117-58e977328dd0 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:27.130062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87d79f7-b4d6-49b3-a117-58e977328dd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:27.130342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87d79f7-b4d6-49b3-a117-58e977328dd0 None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:27.136969 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87d79f7-b4d6-49b3-a117-58e977328dd0 None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:27.138452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87d79f7-b4d6-49b3-a117-58e977328dd0 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:27.140427 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 119/230] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:26 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:27.146616 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87d79f7-b4d6-49b3-a117-58e977328dd0 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:27.245247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc930c7-1eac-44d5-a072-97460b9789ad None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:27.282333 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5acd01-db59-40ed-b4c3-c802dfd32ef4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:27.283322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5acd01-db59-40ed-b4c3-c802dfd32ef4 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:27.283825 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5acd01-db59-40ed-b4c3-c802dfd32ef4 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:27.284325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5acd01-db59-40ed-b4c3-c802dfd32ef4 None admin] PATH_INFO: `/v3/users/1a140a19a341404ca775cbacdb00abf7` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:27.285745 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5acd01-db59-40ed-b4c3-c802dfd32ef4 None admin] RBAC: Authorizing `identity:get_user(user_id=1a140a19a341404ca775cbacdb00abf7)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:27.287402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5acd01-db59-40ed-b4c3-c802dfd32ef4 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:27.289682 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 112/231] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:27 2022] GET /identity/v3/users/1a140a19a341404ca775cbacdb00abf7 => generated 283 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:27.295527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5acd01-db59-40ed-b4c3-c802dfd32ef4 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:27.360555 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:27.375172 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87d79f7-b4d6-49b3-a117-58e977328dd0 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:27.410649 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:27.416862 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a92f96-1370-42da-b7c0-23ab9aa4fede None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:27.418324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a92f96-1370-42da-b7c0-23ab9aa4fede None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:27.418902 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a92f96-1370-42da-b7c0-23ab9aa4fede None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:27.419371 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a92f96-1370-42da-b7c0-23ab9aa4fede None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:27.420311 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a92f96-1370-42da-b7c0-23ab9aa4fede None admin] RBAC: Adding query filter params (role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe, scope.project.id=ea5d616ff88c481baff44c2b3840d3d4, user.id=efe39ad3c10143819e66dd0b1e8eb7e6) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:27.422718 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a92f96-1370-42da-b7c0-23ab9aa4fede None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:27.425814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a92f96-1370-42da-b7c0-23ab9aa4fede None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:27.435785 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 120/232] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:27 2022] GET /identity/v3/role_assignments?user.id=efe39ad3c10143819e66dd0b1e8eb7e6&scope.project.id=ea5d616ff88c481baff44c2b3840d3d4&role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 259 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:27.532431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5acd01-db59-40ed-b4c3-c802dfd32ef4 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:27.593205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac2e051-6e1b-4d47-9335-2de025e6d134 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:27.593870 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac2e051-6e1b-4d47-9335-2de025e6d134 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:27.594147 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac2e051-6e1b-4d47-9335-2de025e6d134 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:27.594417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac2e051-6e1b-4d47-9335-2de025e6d134 None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:27.595569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac2e051-6e1b-4d47-9335-2de025e6d134 None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:27.597171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac2e051-6e1b-4d47-9335-2de025e6d134 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:27.599331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 113/233] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:27 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:27.607256 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a92f96-1370-42da-b7c0-23ab9aa4fede None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:27.720839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a92f96-1370-42da-b7c0-23ab9aa4fede None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:27.759870 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936ee640-40bd-447d-a20d-4b1dd0cf3012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:27.760572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936ee640-40bd-447d-a20d-4b1dd0cf3012 None admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:27.760847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936ee640-40bd-447d-a20d-4b1dd0cf3012 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:27.761118 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936ee640-40bd-447d-a20d-4b1dd0cf3012 None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4/users/1a140a19a341404ca775cbacdb00abf7/roles/fbc5961470b74a85bde6be2171f5f08f` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:27.763522 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936ee640-40bd-447d-a20d-4b1dd0cf3012 None admin] RBAC: Authorizing `identity:create_grant(project_id=ea5d616ff88c481baff44c2b3840d3d4, user_id=1a140a19a341404ca775cbacdb00abf7, role_id=fbc5961470b74a85bde6be2171f5f08f)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:27.764998 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936ee640-40bd-447d-a20d-4b1dd0cf3012 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:27.870027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 121/234] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:04:27 2022] PUT /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4/users/1a140a19a341404ca775cbacdb00abf7/roles/fbc5961470b74a85bde6be2171f5f08f => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:04:28.523554 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179e38a3-9877-48ff-a503-b39a5cd4c579 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:28.524173 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179e38a3-9877-48ff-a503-b39a5cd4c579 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:28.524173 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179e38a3-9877-48ff-a503-b39a5cd4c579 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:28.525563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 114/235] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:28.531538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d593a62-2141-4bee-9497-a0432b493679 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:28.532491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d593a62-2141-4bee-9497-a0432b493679 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:28.533003 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d593a62-2141-4bee-9497-a0432b493679 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:28.567907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-9d593a62-2141-4bee-9497-a0432b493679 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:28.570497 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9d593a62-2141-4bee-9497-a0432b493679 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:28.574501 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:28.685675 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 122/236] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:28.787030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf33c59-dadc-406e-aab5-0452768c85d0 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:28.787559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf33c59-dadc-406e-aab5-0452768c85d0 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:28.788054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf33c59-dadc-406e-aab5-0452768c85d0 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:28.790557 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 115/237] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:28 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:28.801966 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d247998-1cdb-4173-bef6-062757c9d2c9 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:28.802287 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d247998-1cdb-4173-bef6-062757c9d2c9 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:28.802518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d247998-1cdb-4173-bef6-062757c9d2c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:28.827922 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-8d247998-1cdb-4173-bef6-062757c9d2c9 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:28.829242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-8d247998-1cdb-4173-bef6-062757c9d2c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:28.926275 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 123/238] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:29.125713 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf33c59-dadc-406e-aab5-0452768c85d0 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:29.214729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b948b32-8bdc-4fae-8f2b-564e8abc909a None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:29.215076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b948b32-8bdc-4fae-8f2b-564e8abc909a None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:29.215287 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b948b32-8bdc-4fae-8f2b-564e8abc909a None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:29.216983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 124/239] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:29.231654 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ef2a8b-c6fc-4ece-9301-f1e53f69c426 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:29.232162 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ef2a8b-c6fc-4ece-9301-f1e53f69c426 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:29.232630 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ef2a8b-c6fc-4ece-9301-f1e53f69c426 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:29.272525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-03ef2a8b-c6fc-4ece-9301-f1e53f69c426 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:29.277148 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-03ef2a8b-c6fc-4ece-9301-f1e53f69c426 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:29.289362 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf33c59-dadc-406e-aab5-0452768c85d0 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:29.300792 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:29.333965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20470c9c-a37e-4aa9-a0ba-aa381efb107a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:29.334613 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20470c9c-a37e-4aa9-a0ba-aa381efb107a None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:29.334894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20470c9c-a37e-4aa9-a0ba-aa381efb107a None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:29.335167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20470c9c-a37e-4aa9-a0ba-aa381efb107a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:29.348814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20470c9c-a37e-4aa9-a0ba-aa381efb107a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:29.354709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-20470c9c-a37e-4aa9-a0ba-aa381efb107a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:29.354709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:29.354709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:29.354709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:29.354709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:29.354709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:29.354709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:29.354709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:29.354709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:29.354709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:29.354709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:29.354709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:29.354709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:29.354709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:29.354709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:29.354709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:29.355669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:29.355669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:29.355669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:29.355669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:29.355669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:29.355669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:29.355669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:29.355669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:29.355669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:29.355669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:29.355669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:29.355669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:29.355669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:29.355669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:29.355669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:29.355669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:29.357001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:29.357001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:29.357001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:29.357001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:29.357001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:29.357001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:29.357001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:29.357001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:29.357001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:29.357001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:29.357001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:29.357001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:29.357001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 116/240] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:29 2022] GET /identity/v3/roles/reader => generated 84 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:29.357001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20470c9c-a37e-4aa9-a0ba-aa381efb107a None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:29.426111 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 125/241] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:29.426925 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ef2a8b-c6fc-4ece-9301-f1e53f69c426 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:29.432455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:29.519123 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20470c9c-a37e-4aa9-a0ba-aa381efb107a None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:29.588915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ef2a8b-c6fc-4ece-9301-f1e53f69c426 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:29.629126 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d28e36-1684-43c9-b342-6df0769de697 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:29.629782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d28e36-1684-43c9-b342-6df0769de697 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:29.630046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d28e36-1684-43c9-b342-6df0769de697 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:29.630311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d28e36-1684-43c9-b342-6df0769de697 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:29.631791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d28e36-1684-43c9-b342-6df0769de697 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:29.635767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d28e36-1684-43c9-b342-6df0769de697 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:29.639608 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44acf56-abc7-4a6a-93a1-6ddd333079ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:29.640247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44acf56-abc7-4a6a-93a1-6ddd333079ab None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:29.640520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44acf56-abc7-4a6a-93a1-6ddd333079ab None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:29.640784 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44acf56-abc7-4a6a-93a1-6ddd333079ab None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:29.641458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44acf56-abc7-4a6a-93a1-6ddd333079ab None admin] RBAC: Adding query filter params (name=reader) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:29.641754 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44acf56-abc7-4a6a-93a1-6ddd333079ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:29.643171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44acf56-abc7-4a6a-93a1-6ddd333079ab None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:29.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 117/242] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:04:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:29.645319 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d28e36-1684-43c9-b342-6df0769de697 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:29.671675 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 126/243] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:29 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:29.689470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44acf56-abc7-4a6a-93a1-6ddd333079ab None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:29.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:29.763803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:29.823116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44acf56-abc7-4a6a-93a1-6ddd333079ab None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:29.843703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d28e36-1684-43c9-b342-6df0769de697 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:29.871537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6735fb94-fe8f-4828-be99-6882b2de3edd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:29.872905 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6735fb94-fe8f-4828-be99-6882b2de3edd None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:29.873849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6735fb94-fe8f-4828-be99-6882b2de3edd None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:29.874614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6735fb94-fe8f-4828-be99-6882b2de3edd None admin] PATH_INFO: `/v3/users/023bc866d42449b092b1c53d522eb02d` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:29.886890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092a2b0f-e120-4998-9863-98079350de8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:29.887828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092a2b0f-e120-4998-9863-98079350de8b None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:29.888274 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092a2b0f-e120-4998-9863-98079350de8b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:29.888722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092a2b0f-e120-4998-9863-98079350de8b None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:29.889776 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092a2b0f-e120-4998-9863-98079350de8b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:29.892056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092a2b0f-e120-4998-9863-98079350de8b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:29.893858 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6735fb94-fe8f-4828-be99-6882b2de3edd None admin] RBAC: Authorizing `identity:get_user(user_id=023bc866d42449b092b1c53d522eb02d)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:29.895348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6735fb94-fe8f-4828-be99-6882b2de3edd None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:29.897785 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 127/244] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:29 2022] GET /identity/v3/users/023bc866d42449b092b1c53d522eb02d => generated 297 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:29.898402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6735fb94-fe8f-4828-be99-6882b2de3edd None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:29.996364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 118/245] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:04:29 2022] POST /identity/v3/users => generated 293 bytes in 351 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:04:29.998094 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111c6ccb-4fe3-49c1-8e06-022d4c365155 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:29.998631 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111c6ccb-4fe3-49c1-8e06-022d4c365155 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:29.998983 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111c6ccb-4fe3-49c1-8e06-022d4c365155 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:30.000713 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 119/246] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:30.001372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111c6ccb-4fe3-49c1-8e06-022d4c365155 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:30.024988 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6735fb94-fe8f-4828-be99-6882b2de3edd None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:30.048196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:30.073889 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a7c45d-2abd-4b71-8263-6340ceddfbc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:30.074726 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a7c45d-2abd-4b71-8263-6340ceddfbc8 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:30.075098 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a7c45d-2abd-4b71-8263-6340ceddfbc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:30.075472 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a7c45d-2abd-4b71-8263-6340ceddfbc8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:30.088810 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a7c45d-2abd-4b71-8263-6340ceddfbc8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:30.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-c4a7c45d-2abd-4b71-8263-6340ceddfbc8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:30.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:30.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:30.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:30.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:30.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:30.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:30.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:30.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:30.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:30.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:30.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:30.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:30.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:30.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:30.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:30.095261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:30.095261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:30.095261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:30.095261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:30.095261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:30.095261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:30.095261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:30.095261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:30.095261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:30.095261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:30.095261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:30.095261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:30.095261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:30.095261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:30.095261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:30.095261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:30.096319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:30.096319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:30.096319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:30.096319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:30.096319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:30.096319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:30.096319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:30.096319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:30.096319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:30.096319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:30.096319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:30.096319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:30.103196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 128/247] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:04:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:30.104928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74d757c-1bf8-4736-95fd-e325f73558bb None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:30.105340 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74d757c-1bf8-4736-95fd-e325f73558bb None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:30.105715 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74d757c-1bf8-4736-95fd-e325f73558bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:30.125346 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-c74d757c-1bf8-4736-95fd-e325f73558bb None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:30.126785 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c74d757c-1bf8-4736-95fd-e325f73558bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:30.148140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:30.157929 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111c6ccb-4fe3-49c1-8e06-022d4c365155 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:30.215309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46529d2e-29f3-4a64-9200-bc127ab88e89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:30.216404 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46529d2e-29f3-4a64-9200-bc127ab88e89 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:30.217038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46529d2e-29f3-4a64-9200-bc127ab88e89 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:30.217485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46529d2e-29f3-4a64-9200-bc127ab88e89 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:30.241056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 129/248] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:30.241961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74d757c-1bf8-4736-95fd-e325f73558bb None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:30.321826 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46529d2e-29f3-4a64-9200-bc127ab88e89 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:30.323297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46529d2e-29f3-4a64-9200-bc127ab88e89 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:30.426357 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74d757c-1bf8-4736-95fd-e325f73558bb None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:30.439757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:30.475523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e193570-82dd-4157-a093-bfb04cd5957e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:30.476158 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e193570-82dd-4157-a093-bfb04cd5957e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:30.476461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e193570-82dd-4157-a093-bfb04cd5957e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:30.476802 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e193570-82dd-4157-a093-bfb04cd5957e None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:30.477512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e193570-82dd-4157-a093-bfb04cd5957e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:30.477810 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e193570-82dd-4157-a093-bfb04cd5957e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:30.479342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e193570-82dd-4157-a093-bfb04cd5957e None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:30.489359 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 130/249] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:30.492599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:30.496730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e193570-82dd-4157-a093-bfb04cd5957e None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:30.505761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 120/250] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 505 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:30.512396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46529d2e-29f3-4a64-9200-bc127ab88e89 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:30.638321 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e193570-82dd-4157-a093-bfb04cd5957e None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:30.669264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46529d2e-29f3-4a64-9200-bc127ab88e89 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:30.696581 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01f8769-9b55-4439-9834-7712138f460c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:30.697538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01f8769-9b55-4439-9834-7712138f460c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:30.697952 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01f8769-9b55-4439-9834-7712138f460c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:30.698367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01f8769-9b55-4439-9834-7712138f460c None admin] PATH_INFO: `/v3/users/1a140a19a341404ca775cbacdb00abf7` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:30.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01f8769-9b55-4439-9834-7712138f460c None admin] RBAC: Authorizing `identity:get_user(user_id=1a140a19a341404ca775cbacdb00abf7)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:30.702736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01f8769-9b55-4439-9834-7712138f460c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:30.705536 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 131/251] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:30 2022] GET /identity/v3/users/1a140a19a341404ca775cbacdb00abf7 => generated 283 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:30.707381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5beb5c8-d718-47d1-8e82-a6bb92aafa70 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:30.707800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5beb5c8-d718-47d1-8e82-a6bb92aafa70 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:30.708222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5beb5c8-d718-47d1-8e82-a6bb92aafa70 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:30.710660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 132/252] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:30.711896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5beb5c8-d718-47d1-8e82-a6bb92aafa70 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:30.726396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab350bc-581d-427d-a904-02b771ac5542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:30.727589 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab350bc-581d-427d-a904-02b771ac5542 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:30.727897 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab350bc-581d-427d-a904-02b771ac5542 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:30.728202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab350bc-581d-427d-a904-02b771ac5542 None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:30.730249 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab350bc-581d-427d-a904-02b771ac5542 None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:30.732078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab350bc-581d-427d-a904-02b771ac5542 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:30.735076 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 121/253] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:30 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:30.736035 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab350bc-581d-427d-a904-02b771ac5542 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:30.784469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:30.819320 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:30.850450 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5beb5c8-d718-47d1-8e82-a6bb92aafa70 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:30.882088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab350bc-581d-427d-a904-02b771ac5542 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:30.897914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5663a58d-cae1-47d2-901d-6eb958717ef3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:30.899061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5663a58d-cae1-47d2-901d-6eb958717ef3 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:30.900067 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5663a58d-cae1-47d2-901d-6eb958717ef3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:30.900597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5663a58d-cae1-47d2-901d-6eb958717ef3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:30.913996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5663a58d-cae1-47d2-901d-6eb958717ef3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:30.922278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-5663a58d-cae1-47d2-901d-6eb958717ef3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:30.922278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:30.922278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:30.922278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:30.922278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:30.922278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:30.922278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:30.922278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:30.922278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:30.922278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:30.922278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:30.922278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:30.922278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:30.922278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:30.922278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:30.922278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:30.923252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:30.923252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:30.923252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:30.923252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:30.923252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:30.923252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:30.923252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:30.923252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:30.923252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:30.923252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:30.923252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:30.923252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:30.923252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:30.923252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:30.923252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:30.923252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:30.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:30.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:30.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:30.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:30.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:30.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:30.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:30.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:30.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:30.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:30.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:30.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:30.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 133/254] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:30.929763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17aad478-1e00-4f4f-babd-c9bcf677d6a3 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:30.930041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17aad478-1e00-4f4f-babd-c9bcf677d6a3 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:30.930384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17aad478-1e00-4f4f-babd-c9bcf677d6a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:30.932986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e8c5fc-db58-4595-931c-71d80152a1e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:30.934379 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e8c5fc-db58-4595-931c-71d80152a1e3 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:30.934931 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e8c5fc-db58-4595-931c-71d80152a1e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:30.935537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e8c5fc-db58-4595-931c-71d80152a1e3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:30.951894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-17aad478-1e00-4f4f-babd-c9bcf677d6a3 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:30.953373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-17aad478-1e00-4f4f-babd-c9bcf677d6a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:31.033173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 134/255] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:31.034312 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17aad478-1e00-4f4f-babd-c9bcf677d6a3 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:31.072923 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e8c5fc-db58-4595-931c-71d80152a1e3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:31.074413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e8c5fc-db58-4595-931c-71d80152a1e3 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:31.096526 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:31.231244 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:31.287546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17aad478-1e00-4f4f-babd-c9bcf677d6a3 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:31.339504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 122/256] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 604 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:31.352339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e8c5fc-db58-4595-931c-71d80152a1e3 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:31.385108 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64c1ed4-db61-4326-be02-d584d00c41f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:31.385974 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64c1ed4-db61-4326-be02-d584d00c41f9 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:31.386252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64c1ed4-db61-4326-be02-d584d00c41f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:31.386520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64c1ed4-db61-4326-be02-d584d00c41f9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:31.387711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64c1ed4-db61-4326-be02-d584d00c41f9 None admin] RBAC: Adding query filter params (role.id=b5eaad32927541e2a79d4904b5230343, scope.project.id=ea5d616ff88c481baff44c2b3840d3d4, user.id=023bc866d42449b092b1c53d522eb02d) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:31.388020 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64c1ed4-db61-4326-be02-d584d00c41f9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:31.389782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64c1ed4-db61-4326-be02-d584d00c41f9 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:31.404148 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 135/257] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:31 2022] GET /identity/v3/role_assignments?user.id=023bc866d42449b092b1c53d522eb02d&scope.project.id=ea5d616ff88c481baff44c2b3840d3d4&role.id=b5eaad32927541e2a79d4904b5230343 => generated 259 bytes in 364 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:31.404148 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64c1ed4-db61-4326-be02-d584d00c41f9 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:31.538135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:31.549849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64c1ed4-db61-4326-be02-d584d00c41f9 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:31.559168 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e8c5fc-db58-4595-931c-71d80152a1e3 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:31.593768 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0beb8e-9e13-4381-9d82-ea06c65606fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:31.594425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0beb8e-9e13-4381-9d82-ea06c65606fe None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:31.594697 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0beb8e-9e13-4381-9d82-ea06c65606fe None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:31.595116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0beb8e-9e13-4381-9d82-ea06c65606fe None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:31.596539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0beb8e-9e13-4381-9d82-ea06c65606fe None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:31.598041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0beb8e-9e13-4381-9d82-ea06c65606fe None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:31.600151 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 136/258] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:31 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:31.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0beb8e-9e13-4381-9d82-ea06c65606fe None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:31.640249 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86809e12-27e0-4fd7-bd8e-69936726b60e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:31.641046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86809e12-27e0-4fd7-bd8e-69936726b60e None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:31.641320 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86809e12-27e0-4fd7-bd8e-69936726b60e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:31.641589 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86809e12-27e0-4fd7-bd8e-69936726b60e None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:31.642344 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86809e12-27e0-4fd7-bd8e-69936726b60e None admin] RBAC: Adding query filter params (name=member) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:31.642648 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86809e12-27e0-4fd7-bd8e-69936726b60e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:31.644234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86809e12-27e0-4fd7-bd8e-69936726b60e None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:31.660879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 123/259] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:31.661747 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86809e12-27e0-4fd7-bd8e-69936726b60e None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:31.692342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:31.757771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0beb8e-9e13-4381-9d82-ea06c65606fe None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:31.783604 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86809e12-27e0-4fd7-bd8e-69936726b60e None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:31.825184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0863c12f-81a0-4d9d-b0c0-7d3aae580708 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:31.825933 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0863c12f-81a0-4d9d-b0c0-7d3aae580708 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:31.826208 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0863c12f-81a0-4d9d-b0c0-7d3aae580708 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:31.826476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0863c12f-81a0-4d9d-b0c0-7d3aae580708 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:31.827385 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0863c12f-81a0-4d9d-b0c0-7d3aae580708 None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=ea5d616ff88c481baff44c2b3840d3d4, user.id=1a140a19a341404ca775cbacdb00abf7) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:31.827693 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0863c12f-81a0-4d9d-b0c0-7d3aae580708 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:31.829170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0863c12f-81a0-4d9d-b0c0-7d3aae580708 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:31.837830 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df369bf-6497-40b5-bef2-85bb6497ba40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:31.838473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df369bf-6497-40b5-bef2-85bb6497ba40 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:31.838749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df369bf-6497-40b5-bef2-85bb6497ba40 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:31.839153 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df369bf-6497-40b5-bef2-85bb6497ba40 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:31.841151 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 124/260] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:31 2022] GET /identity/v3/role_assignments?user.id=1a140a19a341404ca775cbacdb00abf7&scope.project.id=ea5d616ff88c481baff44c2b3840d3d4&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 608 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:31.842276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0863c12f-81a0-4d9d-b0c0-7d3aae580708 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:31.861264 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df369bf-6497-40b5-bef2-85bb6497ba40 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:31.876795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-3df369bf-6497-40b5-bef2-85bb6497ba40 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:31.876795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:31.876795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:31.876795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:31.876795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:31.876795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:31.876795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:31.876795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:31.876795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:31.876795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:31.876795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:31.876795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:31.876795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:31.876795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:31.876795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:31.876795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:31.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:31.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:31.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:31.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:31.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:31.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:31.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:31.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:31.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:31.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:31.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:31.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:31.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:31.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:31.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:31.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:31.878649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:31.878649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:31.878649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:31.878649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:31.878649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:31.878649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:31.878649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:31.878649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:31.878649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:31.878649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:31.878649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:31.878649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:31.878649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 137/261] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:04:31 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:31.903623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df369bf-6497-40b5-bef2-85bb6497ba40 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:31.952501 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:32.055884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:32.088726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0863c12f-81a0-4d9d-b0c0-7d3aae580708 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:32.131009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df369bf-6497-40b5-bef2-85bb6497ba40 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:32.131219 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02c8edb-f0eb-4429-9e57-31b0f3667bf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:32.131796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02c8edb-f0eb-4429-9e57-31b0f3667bf2 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:32.132069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02c8edb-f0eb-4429-9e57-31b0f3667bf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:32.132366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02c8edb-f0eb-4429-9e57-31b0f3667bf2 None admin] PATH_INFO: `/v3/users/efe39ad3c10143819e66dd0b1e8eb7e6` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:32.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02c8edb-f0eb-4429-9e57-31b0f3667bf2 None admin] RBAC: Authorizing `identity:get_user(user_id=efe39ad3c10143819e66dd0b1e8eb7e6)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:32.135880 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02c8edb-f0eb-4429-9e57-31b0f3667bf2 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:32.137947 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 125/262] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:31 2022] GET /identity/v3/users/efe39ad3c10143819e66dd0b1e8eb7e6 => generated 297 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:32.143287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02c8edb-f0eb-4429-9e57-31b0f3667bf2 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:32.173120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f1e6e9-f629-41c3-891b-b297dfe9dcff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:32.173739 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f1e6e9-f629-41c3-891b-b297dfe9dcff None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:32.174012 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f1e6e9-f629-41c3-891b-b297dfe9dcff None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:32.174563 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f1e6e9-f629-41c3-891b-b297dfe9dcff None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:32.175302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f1e6e9-f629-41c3-891b-b297dfe9dcff None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:32.175600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f1e6e9-f629-41c3-891b-b297dfe9dcff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:32.177759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f1e6e9-f629-41c3-891b-b297dfe9dcff None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:32.189278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 138/263] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:04:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:32.204036 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f1e6e9-f629-41c3-891b-b297dfe9dcff None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:32.326399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:32.334703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02c8edb-f0eb-4429-9e57-31b0f3667bf2 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:32.349107 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f1e6e9-f629-41c3-891b-b297dfe9dcff None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:32.404132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9c4857-f495-4fc7-beb7-af02cdc77f3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:32.404800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9c4857-f495-4fc7-beb7-af02cdc77f3e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:32.405075 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9c4857-f495-4fc7-beb7-af02cdc77f3e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:32.405345 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9c4857-f495-4fc7-beb7-af02cdc77f3e None admin] PATH_INFO: `/v3/users/1a140a19a341404ca775cbacdb00abf7` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:32.407036 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9c4857-f495-4fc7-beb7-af02cdc77f3e None admin] RBAC: Authorizing `identity:get_user(user_id=1a140a19a341404ca775cbacdb00abf7)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:32.409047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9c4857-f495-4fc7-beb7-af02cdc77f3e None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:32.411011 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 139/264] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:32 2022] GET /identity/v3/users/1a140a19a341404ca775cbacdb00abf7 => generated 283 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:32.411768 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f046d67b-f10c-49ae-ab09-383e2797c0c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:32.412443 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f046d67b-f10c-49ae-ab09-383e2797c0c3 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:32.412722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f046d67b-f10c-49ae-ab09-383e2797c0c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:32.412990 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f046d67b-f10c-49ae-ab09-383e2797c0c3 None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:32.420936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f046d67b-f10c-49ae-ab09-383e2797c0c3 None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:32.421207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9c4857-f495-4fc7-beb7-af02cdc77f3e None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:32.422402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f046d67b-f10c-49ae-ab09-383e2797c0c3 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:32.426202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 126/265] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:32 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:32.432332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f046d67b-f10c-49ae-ab09-383e2797c0c3 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:32.486582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:32.639175 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f046d67b-f10c-49ae-ab09-383e2797c0c3 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:32.658377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9c4857-f495-4fc7-beb7-af02cdc77f3e None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:32.684683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c3ea2b-1256-4583-b245-22af99db2aa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:32.685367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c3ea2b-1256-4583-b245-22af99db2aa3 None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:32.685646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c3ea2b-1256-4583-b245-22af99db2aa3 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:32.685919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c3ea2b-1256-4583-b245-22af99db2aa3 None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4/users/efe39ad3c10143819e66dd0b1e8eb7e6/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:32.688163 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c3ea2b-1256-4583-b245-22af99db2aa3 None admin] RBAC: Authorizing `identity:create_grant(project_id=ea5d616ff88c481baff44c2b3840d3d4, user_id=efe39ad3c10143819e66dd0b1e8eb7e6, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:32.690646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c3ea2b-1256-4583-b245-22af99db2aa3 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:32.724852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d421c679-6f03-4c27-802c-3a1c6fb7f781 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:32.725532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d421c679-6f03-4c27-802c-3a1c6fb7f781 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:32.725815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d421c679-6f03-4c27-802c-3a1c6fb7f781 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:32.726088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d421c679-6f03-4c27-802c-3a1c6fb7f781 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:32.748741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:32.779647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 127/266] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:04:32 2022] PUT /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4/users/efe39ad3c10143819e66dd0b1e8eb7e6/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:04:32.781150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1cd997-9542-46a6-ba68-dc6708ee4f22 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:32.781446 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1cd997-9542-46a6-ba68-dc6708ee4f22 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:32.781747 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1cd997-9542-46a6-ba68-dc6708ee4f22 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:32.783689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 128/267] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:32.785084 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e71474-23ab-40d2-a784-f9555477c094 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:32.785372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e71474-23ab-40d2-a784-f9555477c094 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:32.785554 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e71474-23ab-40d2-a784-f9555477c094 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:32.787325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 129/268] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:32.796163 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35f2098-4fc7-4a01-b67e-cb6e435bc3d8 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:32.796163 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35f2098-4fc7-4a01-b67e-cb6e435bc3d8 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:32.796163 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35f2098-4fc7-4a01-b67e-cb6e435bc3d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:32.819609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-a35f2098-4fc7-4a01-b67e-cb6e435bc3d8 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:32.820970 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a35f2098-4fc7-4a01-b67e-cb6e435bc3d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:32.835297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d421c679-6f03-4c27-802c-3a1c6fb7f781 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:32.837605 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d421c679-6f03-4c27-802c-3a1c6fb7f781 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:32.910889 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 130/269] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:32.912203 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0fda26-69b3-431d-a4d4-80af5da9e592 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:32.912490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0fda26-69b3-431d-a4d4-80af5da9e592 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:32.912758 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0fda26-69b3-431d-a4d4-80af5da9e592 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:32.938370 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-bf0fda26-69b3-431d-a4d4-80af5da9e592 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:32.939646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-bf0fda26-69b3-431d-a4d4-80af5da9e592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:32.949808 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:32.994139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 140/270] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 573 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:33.000652 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d421c679-6f03-4c27-802c-3a1c6fb7f781 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:33.051628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 131/271] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:33.084544 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:33.156224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d421c679-6f03-4c27-802c-3a1c6fb7f781 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:33.208446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458b3930-7891-41c0-b193-fe1c6e7cfe2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:33.209142 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458b3930-7891-41c0-b193-fe1c6e7cfe2b None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:33.209471 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458b3930-7891-41c0-b193-fe1c6e7cfe2b None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:33.209753 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458b3930-7891-41c0-b193-fe1c6e7cfe2b None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:33.216969 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458b3930-7891-41c0-b193-fe1c6e7cfe2b None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:33.218539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458b3930-7891-41c0-b193-fe1c6e7cfe2b None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:33.220375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 141/272] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:32 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:33.226601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0fda26-69b3-431d-a4d4-80af5da9e592 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:33.294054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458b3930-7891-41c0-b193-fe1c6e7cfe2b None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:33.419381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458b3930-7891-41c0-b193-fe1c6e7cfe2b None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:33.426123 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0fda26-69b3-431d-a4d4-80af5da9e592 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:33.443381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:33.465228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76df91f1-3a4b-4b2e-a93b-fe723d237c44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:33.466231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76df91f1-3a4b-4b2e-a93b-fe723d237c44 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:33.466983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76df91f1-3a4b-4b2e-a93b-fe723d237c44 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:33.467885 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76df91f1-3a4b-4b2e-a93b-fe723d237c44 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:33.469692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbab9ce-908a-42b1-8cd2-cb2058fbed9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:33.470310 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbab9ce-908a-42b1-8cd2-cb2058fbed9c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:33.470586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbab9ce-908a-42b1-8cd2-cb2058fbed9c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:33.470853 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbab9ce-908a-42b1-8cd2-cb2058fbed9c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:33.471588 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbab9ce-908a-42b1-8cd2-cb2058fbed9c None admin] RBAC: Adding query filter params (role.id=fbc5961470b74a85bde6be2171f5f08f, scope.project.id=ea5d616ff88c481baff44c2b3840d3d4, user.id=1a140a19a341404ca775cbacdb00abf7) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:33.471892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbab9ce-908a-42b1-8cd2-cb2058fbed9c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:33.473614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbab9ce-908a-42b1-8cd2-cb2058fbed9c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:33.486103 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 132/273] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:33 2022] GET /identity/v3/role_assignments?user.id=1a140a19a341404ca775cbacdb00abf7&scope.project.id=ea5d616ff88c481baff44c2b3840d3d4&role.id=fbc5961470b74a85bde6be2171f5f08f => generated 608 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:33.487311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbab9ce-908a-42b1-8cd2-cb2058fbed9c None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:33.501217 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76df91f1-3a4b-4b2e-a93b-fe723d237c44 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:33.520507 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:33.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-76df91f1-3a4b-4b2e-a93b-fe723d237c44 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:33.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:33.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:33.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:33.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:33.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:33.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:33.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:33.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:33.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:33.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:33.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:33.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:33.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:33.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:33.520824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:33.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:33.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:33.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:33.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:33.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:33.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:33.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:33.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:33.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:33.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:33.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:33.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:33.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:33.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:33.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:33.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:33.522495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:33.522495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:33.522495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:33.522495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:33.522495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:33.522495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:33.522495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:33.522495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:33.522495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:33.522495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:33.522495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:33.522495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:33.522495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 142/274] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:04:33.533911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76df91f1-3a4b-4b2e-a93b-fe723d237c44 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:33.648269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbab9ce-908a-42b1-8cd2-cb2058fbed9c None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:33.688401 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e612592a-a08e-4816-b457-cf32df69c431 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:33.689064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e612592a-a08e-4816-b457-cf32df69c431 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:33.689342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e612592a-a08e-4816-b457-cf32df69c431 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:33.689616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e612592a-a08e-4816-b457-cf32df69c431 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:33.690812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e612592a-a08e-4816-b457-cf32df69c431 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:33.694786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e612592a-a08e-4816-b457-cf32df69c431 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:33.696822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 133/275] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:04:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:33.720506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e612592a-a08e-4816-b457-cf32df69c431 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:33.753239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76df91f1-3a4b-4b2e-a93b-fe723d237c44 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:33.794977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22b3baa-5c37-456e-8a44-3587139c3e46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:33.796296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22b3baa-5c37-456e-8a44-3587139c3e46 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:33.796559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22b3baa-5c37-456e-8a44-3587139c3e46 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:33.797048 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22b3baa-5c37-456e-8a44-3587139c3e46 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:33.798120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22b3baa-5c37-456e-8a44-3587139c3e46 None admin] RBAC: Adding query filter params (name=member) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:33.798520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22b3baa-5c37-456e-8a44-3587139c3e46 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:33.799985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22b3baa-5c37-456e-8a44-3587139c3e46 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:33.812143 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 143/276] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:33.821466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22b3baa-5c37-456e-8a44-3587139c3e46 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:33.837664 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:33.870859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e612592a-a08e-4816-b457-cf32df69c431 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:33.874349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:33.911879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906cee44-22d1-421f-97c7-2f028c2ebcb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:33.913009 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906cee44-22d1-421f-97c7-2f028c2ebcb2 None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:33.913449 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906cee44-22d1-421f-97c7-2f028c2ebcb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:33.914009 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906cee44-22d1-421f-97c7-2f028c2ebcb2 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:33.915353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906cee44-22d1-421f-97c7-2f028c2ebcb2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:33.917133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906cee44-22d1-421f-97c7-2f028c2ebcb2 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:33.943660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22b3baa-5c37-456e-8a44-3587139c3e46 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:33.991675 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a053da-c1c4-47eb-8395-b2f18a410d5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:33.992287 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a053da-c1c4-47eb-8395-b2f18a410d5b None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:33.993164 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a053da-c1c4-47eb-8395-b2f18a410d5b None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:33.993498 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a053da-c1c4-47eb-8395-b2f18a410d5b None admin] PATH_INFO: `/v3/users/5eeda698b69748d0ba75340d340cc330` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:34.008792 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a053da-c1c4-47eb-8395-b2f18a410d5b None admin] RBAC: Authorizing `identity:get_user(user_id=5eeda698b69748d0ba75340d340cc330)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:34.010255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a053da-c1c4-47eb-8395-b2f18a410d5b None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:34.011995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 144/277] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:33 2022] GET /identity/v3/users/5eeda698b69748d0ba75340d340cc330 => generated 293 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:34.013268 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9421de49-4079-40f6-873c-cd7995086b42 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:34.013537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9421de49-4079-40f6-873c-cd7995086b42 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:34.013805 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9421de49-4079-40f6-873c-cd7995086b42 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:34.015159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 145/278] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:34 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:34.019815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9421de49-4079-40f6-873c-cd7995086b42 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:34.028209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 134/279] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:04:33 2022] POST /identity/v3/users => generated 293 bytes in 308 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:04:34.029624 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525e98af-439d-49ed-8211-95f4d1a62554 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:34.030406 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525e98af-439d-49ed-8211-95f4d1a62554 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:34.031078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525e98af-439d-49ed-8211-95f4d1a62554 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:34.055820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-525e98af-439d-49ed-8211-95f4d1a62554 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:34.057192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-525e98af-439d-49ed-8211-95f4d1a62554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:34.161671 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 135/280] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:34.173192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:34.179492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9421de49-4079-40f6-873c-cd7995086b42 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:34.221649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10883dc0-b0f5-4533-abff-1735c8b3a5b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:34.222256 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10883dc0-b0f5-4533-abff-1735c8b3a5b0 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:34.222590 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10883dc0-b0f5-4533-abff-1735c8b3a5b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:34.222856 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10883dc0-b0f5-4533-abff-1735c8b3a5b0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:34.223608 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10883dc0-b0f5-4533-abff-1735c8b3a5b0 None admin] RBAC: Adding query filter params (role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe, user.id=5eeda698b69748d0ba75340d340cc330, scope.system=all) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:34.223996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10883dc0-b0f5-4533-abff-1735c8b3a5b0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:34.225454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10883dc0-b0f5-4533-abff-1735c8b3a5b0 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:34.233049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 146/281] 10.222.0.47 () {56 vars in 1308 bytes} [Sun Aug 21 01:04:34 2022] GET /identity/v3/role_assignments?user.id=5eeda698b69748d0ba75340d340cc330&scope.system=all&role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 226 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:34.464408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525e98af-439d-49ed-8211-95f4d1a62554 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:34.546958 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:34.605553 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525e98af-439d-49ed-8211-95f4d1a62554 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:34.661774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0387ba27-0056-45f2-a1da-dba208ff5d3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:34.662432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0387ba27-0056-45f2-a1da-dba208ff5d3c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:34.662702 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0387ba27-0056-45f2-a1da-dba208ff5d3c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:34.662967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0387ba27-0056-45f2-a1da-dba208ff5d3c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:34.678929 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0387ba27-0056-45f2-a1da-dba208ff5d3c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:34.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-0387ba27-0056-45f2-a1da-dba208ff5d3c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:34.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:34.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:34.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:34.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:34.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:34.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:34.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:34.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:34.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:34.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:34.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:34.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:34.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:34.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:34.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:34.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:34.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:34.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:34.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:34.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:34.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:34.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:34.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:34.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:34.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:34.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:34.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:34.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:34.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:34.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:34.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:34.686476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:34.686476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:34.686476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:34.686476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:34.686476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:34.686476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:34.686476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:34.686476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:34.686476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:34.686476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:34.686476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:34.686476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:34.686476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 136/282] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:34 2022] GET /identity/v3/roles/reader => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:04:34.691865 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10883dc0-b0f5-4533-abff-1735c8b3a5b0 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:34.832878 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10883dc0-b0f5-4533-abff-1735c8b3a5b0 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:34.879842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee271f0-b519-4279-aedb-6fcf7cec8022 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:34.880741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee271f0-b519-4279-aedb-6fcf7cec8022 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:34.880741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee271f0-b519-4279-aedb-6fcf7cec8022 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:34.880983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee271f0-b519-4279-aedb-6fcf7cec8022 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:34.881582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee271f0-b519-4279-aedb-6fcf7cec8022 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:34.881878 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee271f0-b519-4279-aedb-6fcf7cec8022 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:34.883373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee271f0-b519-4279-aedb-6fcf7cec8022 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:34.893150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 147/283] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:34 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:34.899224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0387ba27-0056-45f2-a1da-dba208ff5d3c None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:35.029032 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0387ba27-0056-45f2-a1da-dba208ff5d3c None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:35.053460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:35.077251 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a0d0d3-0fc5-497e-88a0-44fc9d1c3d0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:35.077899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a0d0d3-0fc5-497e-88a0-44fc9d1c3d0e None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:35.078174 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a0d0d3-0fc5-497e-88a0-44fc9d1c3d0e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:35.078452 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a0d0d3-0fc5-497e-88a0-44fc9d1c3d0e None admin] PATH_INFO: `/v3/users/023bc866d42449b092b1c53d522eb02d` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:35.081571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a0d0d3-0fc5-497e-88a0-44fc9d1c3d0e None admin] RBAC: Authorizing `identity:get_user(user_id=023bc866d42449b092b1c53d522eb02d)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:35.083027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a0d0d3-0fc5-497e-88a0-44fc9d1c3d0e None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:35.084672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 137/284] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:34 2022] GET /identity/v3/users/023bc866d42449b092b1c53d522eb02d => generated 297 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:35.090332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee271f0-b519-4279-aedb-6fcf7cec8022 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:35.124891 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:35.226060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee271f0-b519-4279-aedb-6fcf7cec8022 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:35.278364 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64c1fe4-8c4f-44d6-9dbd-ecd793473f7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:35.279017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64c1fe4-8c4f-44d6-9dbd-ecd793473f7d None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:35.279293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64c1fe4-8c4f-44d6-9dbd-ecd793473f7d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:35.279563 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64c1fe4-8c4f-44d6-9dbd-ecd793473f7d None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:35.280757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64c1fe4-8c4f-44d6-9dbd-ecd793473f7d None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:35.282225 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64c1fe4-8c4f-44d6-9dbd-ecd793473f7d None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:35.284262 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 148/285] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:35 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:35.291375 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a0d0d3-0fc5-497e-88a0-44fc9d1c3d0e None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:35.301925 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34221b12-430b-43ed-b630-0947b942f197 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:35.302240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34221b12-430b-43ed-b630-0947b942f197 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:35.302490 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34221b12-430b-43ed-b630-0947b942f197 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:35.306314 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 149/286] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:35 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:35.312062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c841d7-e78d-4e69-9029-87c429335d09 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:35.312373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c841d7-e78d-4e69-9029-87c429335d09 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:35.312648 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c841d7-e78d-4e69-9029-87c429335d09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:35.353693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-c3c841d7-e78d-4e69-9029-87c429335d09 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:35.360145 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c3c841d7-e78d-4e69-9029-87c429335d09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:35.459067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a0d0d3-0fc5-497e-88a0-44fc9d1c3d0e None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:35.498845 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b14a2b-bbb0-4f30-b23d-1a039e812849 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:35.500712 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b14a2b-bbb0-4f30-b23d-1a039e812849 None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:35.501867 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b14a2b-bbb0-4f30-b23d-1a039e812849 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:35.502513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b14a2b-bbb0-4f30-b23d-1a039e812849 None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4/users/023bc866d42449b092b1c53d522eb02d/roles/b5eaad32927541e2a79d4904b5230343` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:35.506690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b14a2b-bbb0-4f30-b23d-1a039e812849 None admin] RBAC: Authorizing `identity:create_grant(project_id=ea5d616ff88c481baff44c2b3840d3d4, user_id=023bc866d42449b092b1c53d522eb02d, role_id=b5eaad32927541e2a79d4904b5230343)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:35.509621 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b14a2b-bbb0-4f30-b23d-1a039e812849 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:35.520521 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 150/287] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:35.605443 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 138/288] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:04:35 2022] PUT /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4/users/023bc866d42449b092b1c53d522eb02d/roles/b5eaad32927541e2a79d4904b5230343 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:04:35.943127 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c841d7-e78d-4e69-9029-87c429335d09 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:35.970658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:36.074907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c841d7-e78d-4e69-9029-87c429335d09 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:36.126024 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54125a1-0199-4592-8e60-520fbaac1889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:36.127014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54125a1-0199-4592-8e60-520fbaac1889 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:36.127489 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54125a1-0199-4592-8e60-520fbaac1889 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:36.127970 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54125a1-0199-4592-8e60-520fbaac1889 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:36.148395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54125a1-0199-4592-8e60-520fbaac1889 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:36.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-d54125a1-0199-4592-8e60-520fbaac1889 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:36.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:36.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:36.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:36.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:36.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:36.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:36.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:36.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:36.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:36.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:36.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:36.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:36.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:36.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:36.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:36.155119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:36.155119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:36.155119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:36.155119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:36.155119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:36.155119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:36.155119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:36.155119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:36.155119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:36.155119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:36.155119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:36.155119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:36.155119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:36.155119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:36.155119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:36.155119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:36.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:36.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:36.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:36.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:36.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:36.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:36.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:36.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:36.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:36.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:36.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:36.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:36.158495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 151/289] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:36.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b14a2b-bbb0-4f30-b23d-1a039e812849 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:36.239225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:36.323012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b14a2b-bbb0-4f30-b23d-1a039e812849 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:36.391829 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d98070a-d184-4d66-a6d6-4e6ebf59409c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:36.392475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d98070a-d184-4d66-a6d6-4e6ebf59409c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:36.392752 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d98070a-d184-4d66-a6d6-4e6ebf59409c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:36.393017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d98070a-d184-4d66-a6d6-4e6ebf59409c None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:36.393685 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d98070a-d184-4d66-a6d6-4e6ebf59409c None admin] RBAC: Adding query filter params (name=member) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:36.393979 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d98070a-d184-4d66-a6d6-4e6ebf59409c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:36.395345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d98070a-d184-4d66-a6d6-4e6ebf59409c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:36.405704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 139/290] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:36 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:36.411101 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54125a1-0199-4592-8e60-520fbaac1889 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:36.503243 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:36.524877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54125a1-0199-4592-8e60-520fbaac1889 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:36.549702 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ed7200-35f1-454c-bf66-d9a90ef02fbe None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:36.550139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ed7200-35f1-454c-bf66-d9a90ef02fbe None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:36.550311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ed7200-35f1-454c-bf66-d9a90ef02fbe None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:36.551663 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 140/291] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:04:36.557836 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421ea15a-7a23-4f83-8edf-a80ef14d22a0 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:36.558187 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421ea15a-7a23-4f83-8edf-a80ef14d22a0 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:36.558398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421ea15a-7a23-4f83-8edf-a80ef14d22a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:36.583897 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-421ea15a-7a23-4f83-8edf-a80ef14d22a0 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:36.584765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37110d7-9deb-452a-93fc-3744e96cafc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:36.585282 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-421ea15a-7a23-4f83-8edf-a80ef14d22a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:36.585479 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37110d7-9deb-452a-93fc-3744e96cafc3 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:36.585692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37110d7-9deb-452a-93fc-3744e96cafc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:36.586482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37110d7-9deb-452a-93fc-3744e96cafc3 None admin] PATH_INFO: `/v3/users/efe39ad3c10143819e66dd0b1e8eb7e6` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:36.587683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37110d7-9deb-452a-93fc-3744e96cafc3 None admin] RBAC: Authorizing `identity:get_user(user_id=efe39ad3c10143819e66dd0b1e8eb7e6)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:36.589340 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37110d7-9deb-452a-93fc-3744e96cafc3 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:36.591255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 152/292] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:36 2022] GET /identity/v3/users/efe39ad3c10143819e66dd0b1e8eb7e6 => generated 297 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:36.597209 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37110d7-9deb-452a-93fc-3744e96cafc3 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:36.695965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 141/293] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:36.767653 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37110d7-9deb-452a-93fc-3744e96cafc3 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:36.818150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a939df-2c64-411f-88bb-0602333d8883 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:36.819349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a939df-2c64-411f-88bb-0602333d8883 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:36.819769 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a939df-2c64-411f-88bb-0602333d8883 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:36.820211 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a939df-2c64-411f-88bb-0602333d8883 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:36.847692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dee027-890c-44bc-9223-e3d4a70fde66 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:36.847692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dee027-890c-44bc-9223-e3d4a70fde66 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:36.847692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dee027-890c-44bc-9223-e3d4a70fde66 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:36.847692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 142/294] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:36.859696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e2f87e-381f-462a-9faa-4462e6b221f8 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:36.860016 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e2f87e-381f-462a-9faa-4462e6b221f8 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:36.860307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e2f87e-381f-462a-9faa-4462e6b221f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:36.862018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:36.913211 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-e0e2f87e-381f-462a-9faa-4462e6b221f8 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:36.914973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e0e2f87e-381f-462a-9faa-4462e6b221f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:36.919212 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a939df-2c64-411f-88bb-0602333d8883 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:36.921035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a939df-2c64-411f-88bb-0602333d8883 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:37.018373 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:37.043083 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 153/295] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 446 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:37.049775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14cad74-1c11-4ae2-879d-9598dbb9e013 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:37.050407 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14cad74-1c11-4ae2-879d-9598dbb9e013 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:37.050914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14cad74-1c11-4ae2-879d-9598dbb9e013 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:37.052855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 154/296] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:37 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:37.053935 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14cad74-1c11-4ae2-879d-9598dbb9e013 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:37.081082 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 143/297] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:04:37.082364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6c3d35-e93c-4109-b3ad-13ba362248c3 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:37.082646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6c3d35-e93c-4109-b3ad-13ba362248c3 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:37.082915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6c3d35-e93c-4109-b3ad-13ba362248c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:37.109113 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-0f6c3d35-e93c-4109-b3ad-13ba362248c3 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:37.110916 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-0f6c3d35-e93c-4109-b3ad-13ba362248c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:37.124627 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:37.195006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 144/298] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:37.195006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6c3d35-e93c-4109-b3ad-13ba362248c3 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:37.209754 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14cad74-1c11-4ae2-879d-9598dbb9e013 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:37.259293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67aa73e9-402b-4705-bc12-80bf7d9ba91c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:37.260220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67aa73e9-402b-4705-bc12-80bf7d9ba91c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:37.260728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67aa73e9-402b-4705-bc12-80bf7d9ba91c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:37.261145 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67aa73e9-402b-4705-bc12-80bf7d9ba91c None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:37.262695 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67aa73e9-402b-4705-bc12-80bf7d9ba91c None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:37.264436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67aa73e9-402b-4705-bc12-80bf7d9ba91c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:37.272215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 155/299] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:37 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:37.274137 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67aa73e9-402b-4705-bc12-80bf7d9ba91c None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:37.340489 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:37.355520 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6c3d35-e93c-4109-b3ad-13ba362248c3 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:37.395628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d96e3c-a9be-4e3f-90d1-f4c724963167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:37.396664 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d96e3c-a9be-4e3f-90d1-f4c724963167 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:37.397177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d96e3c-a9be-4e3f-90d1-f4c724963167 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:37.397646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d96e3c-a9be-4e3f-90d1-f4c724963167 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:37.410846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d96e3c-a9be-4e3f-90d1-f4c724963167 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:37.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-b5d96e3c-a9be-4e3f-90d1-f4c724963167 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:37.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:37.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:37.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:37.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:37.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:37.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:37.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:37.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:37.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:37.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:37.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:37.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:37.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:37.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:37.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:37.418806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:37.418806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:37.418806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:37.418806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:37.418806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:37.418806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:37.418806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:37.418806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:37.418806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:37.418806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:37.418806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:37.418806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:37.418806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:37.418806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:37.418806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:37.418806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:37.419595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:37.419595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:37.419595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:37.419595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:37.419595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:37.419595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:37.419595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:37.419595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:37.419595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:37.419595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:37.419595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:37.419595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:37.422367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 145/300] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:37.433167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d96e3c-a9be-4e3f-90d1-f4c724963167 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:37.471151 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67aa73e9-402b-4705-bc12-80bf7d9ba91c None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:37.493234 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:37.515360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfe1847-59a1-41de-8db0-8d6b6040ff89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:37.515971 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfe1847-59a1-41de-8db0-8d6b6040ff89 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:37.516268 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfe1847-59a1-41de-8db0-8d6b6040ff89 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:37.516536 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfe1847-59a1-41de-8db0-8d6b6040ff89 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:37.517317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfe1847-59a1-41de-8db0-8d6b6040ff89 None admin] RBAC: Adding query filter params (role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe, scope.project.id=ea5d616ff88c481baff44c2b3840d3d4, user.id=efe39ad3c10143819e66dd0b1e8eb7e6) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:37.517623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfe1847-59a1-41de-8db0-8d6b6040ff89 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:37.519814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfe1847-59a1-41de-8db0-8d6b6040ff89 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:37.529815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 156/301] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:37 2022] GET /identity/v3/role_assignments?user.id=efe39ad3c10143819e66dd0b1e8eb7e6&scope.project.id=ea5d616ff88c481baff44c2b3840d3d4&role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 608 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:37.530597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfe1847-59a1-41de-8db0-8d6b6040ff89 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:37.592517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d96e3c-a9be-4e3f-90d1-f4c724963167 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:37.637007 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3d7438-ea87-43b7-8a20-1c0d33479424 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:37.638311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3d7438-ea87-43b7-8a20-1c0d33479424 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:37.639180 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3d7438-ea87-43b7-8a20-1c0d33479424 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:37.639512 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3d7438-ea87-43b7-8a20-1c0d33479424 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:37.640298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3d7438-ea87-43b7-8a20-1c0d33479424 None admin] RBAC: Adding query filter params (name=member) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:37.640668 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3d7438-ea87-43b7-8a20-1c0d33479424 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:37.642275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3d7438-ea87-43b7-8a20-1c0d33479424 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:37.654067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 146/302] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:37.655188 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3d7438-ea87-43b7-8a20-1c0d33479424 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:37.698009 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:37.744521 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfe1847-59a1-41de-8db0-8d6b6040ff89 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:37.796823 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c657d74a-999e-45f6-9745-2e7ccb6fbc93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:37.797465 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c657d74a-999e-45f6-9745-2e7ccb6fbc93 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:37.797733 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c657d74a-999e-45f6-9745-2e7ccb6fbc93 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:37.798056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c657d74a-999e-45f6-9745-2e7ccb6fbc93 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:37.802139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c657d74a-999e-45f6-9745-2e7ccb6fbc93 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:37.802139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c657d74a-999e-45f6-9745-2e7ccb6fbc93 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:37.808771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 157/303] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:04:37 2022] GET /identity/v3/domains/default => generated 199 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:37.808771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c657d74a-999e-45f6-9745-2e7ccb6fbc93 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:37.822992 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3d7438-ea87-43b7-8a20-1c0d33479424 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:37.865950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25dc760-165a-4cdc-93c1-6ccb446c8402 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:37.866724 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25dc760-165a-4cdc-93c1-6ccb446c8402 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:37.867002 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25dc760-165a-4cdc-93c1-6ccb446c8402 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:37.867795 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25dc760-165a-4cdc-93c1-6ccb446c8402 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:37.871388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:37.880288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25dc760-165a-4cdc-93c1-6ccb446c8402 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:37.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-e25dc760-165a-4cdc-93c1-6ccb446c8402 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:37.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:37.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:37.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:37.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:37.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:37.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:37.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:37.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:37.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:37.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:37.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:37.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:37.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:37.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:37.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:37.886800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:37.886800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:37.886800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:37.886800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:37.886800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:37.886800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:37.886800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:37.886800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:37.886800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:37.886800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:37.886800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:37.886800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:37.886800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:37.886800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:37.886800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:37.886800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:37.887617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:37.887617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:37.887617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:37.887617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:37.887617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:37.887617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:37.887617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:37.887617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:37.887617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:37.887617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:37.887617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:37.887617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:37.887617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 147/304] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:37 2022] GET /identity/v3/roles/reader => generated 84 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:37.888828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25dc760-165a-4cdc-93c1-6ccb446c8402 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:37.934790 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c657d74a-999e-45f6-9745-2e7ccb6fbc93 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:37.990571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c36193-48f1-4f45-a61f-3eb43efa6b3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:37.991287 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c36193-48f1-4f45-a61f-3eb43efa6b3a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:37.991667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c36193-48f1-4f45-a61f-3eb43efa6b3a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:37.992732 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c36193-48f1-4f45-a61f-3eb43efa6b3a None admin] PATH_INFO: `/v3/users/5eeda698b69748d0ba75340d340cc330` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:37.994547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c36193-48f1-4f45-a61f-3eb43efa6b3a None admin] RBAC: Authorizing `identity:get_user(user_id=5eeda698b69748d0ba75340d340cc330)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:37.997092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c36193-48f1-4f45-a61f-3eb43efa6b3a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:37.999836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 158/305] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:37 2022] GET /identity/v3/users/5eeda698b69748d0ba75340d340cc330 => generated 293 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:38.004545 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c36193-48f1-4f45-a61f-3eb43efa6b3a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:38.011275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:38.031070 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25dc760-165a-4cdc-93c1-6ccb446c8402 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:38.074472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1ca311-7b93-42a9-a0da-675959caa2cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:38.074992 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1ca311-7b93-42a9-a0da-675959caa2cf None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:38.075267 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1ca311-7b93-42a9-a0da-675959caa2cf None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:38.075532 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1ca311-7b93-42a9-a0da-675959caa2cf None admin] PATH_INFO: `/v3/groups` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:38.076176 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1ca311-7b93-42a9-a0da-675959caa2cf None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:38.077605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1ca311-7b93-42a9-a0da-675959caa2cf None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:38.123339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c36193-48f1-4f45-a61f-3eb43efa6b3a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:38.141282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:38.148302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 148/306] 10.222.0.47 () {60 vars in 1134 bytes} [Sun Aug 21 01:04:37 2022] POST /identity/v3/groups => generated 229 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:04:38.149102 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1ca311-7b93-42a9-a0da-675959caa2cf None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:38.159199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389a3a26-5d3d-4349-bad8-a9941d34e8e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:38.159836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389a3a26-5d3d-4349-bad8-a9941d34e8e5 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:38.160213 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389a3a26-5d3d-4349-bad8-a9941d34e8e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:38.160491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389a3a26-5d3d-4349-bad8-a9941d34e8e5 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:38.161187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389a3a26-5d3d-4349-bad8-a9941d34e8e5 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:38.161489 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389a3a26-5d3d-4349-bad8-a9941d34e8e5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:38.162952 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389a3a26-5d3d-4349-bad8-a9941d34e8e5 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:38.173995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 159/307] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:37 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:38.180451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83cc408-f16d-481a-bcb9-d7669d4c6576 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:38.180767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83cc408-f16d-481a-bcb9-d7669d4c6576 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:38.181008 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83cc408-f16d-481a-bcb9-d7669d4c6576 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:38.182597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 160/308] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:38.183199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83cc408-f16d-481a-bcb9-d7669d4c6576 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:38.281687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1ca311-7b93-42a9-a0da-675959caa2cf None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:38.309106 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83cc408-f16d-481a-bcb9-d7669d4c6576 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:38.328332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8e130b-e9e1-4b3e-a50d-4fbdfd6a6ed8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:38.329549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8e130b-e9e1-4b3e-a50d-4fbdfd6a6ed8 None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:38.329549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8e130b-e9e1-4b3e-a50d-4fbdfd6a6ed8 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:38.329549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8e130b-e9e1-4b3e-a50d-4fbdfd6a6ed8 None admin] PATH_INFO: `/v3/system/users/5eeda698b69748d0ba75340d340cc330/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:38.331735 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8e130b-e9e1-4b3e-a50d-4fbdfd6a6ed8 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=5eeda698b69748d0ba75340d340cc330, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:38.334277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8e130b-e9e1-4b3e-a50d-4fbdfd6a6ed8 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:38.356763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbc8370-8a94-49b1-84ce-616f0bee7a3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:38.357511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbc8370-8a94-49b1-84ce-616f0bee7a3c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:38.357874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbc8370-8a94-49b1-84ce-616f0bee7a3c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:38.358212 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbc8370-8a94-49b1-84ce-616f0bee7a3c None admin] PATH_INFO: `/v3/users/79047ab97cfc4a83a1a05a84c845c021` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:38.373615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbc8370-8a94-49b1-84ce-616f0bee7a3c None admin] RBAC: Authorizing `identity:get_user(user_id=79047ab97cfc4a83a1a05a84c845c021)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:38.375118 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbc8370-8a94-49b1-84ce-616f0bee7a3c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:38.377074 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 161/309] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:38 2022] GET /identity/v3/users/79047ab97cfc4a83a1a05a84c845c021 => generated 293 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:38.378109 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff533aa4-723d-4f48-a4d5-259dca6fb45f None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:38.378389 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff533aa4-723d-4f48-a4d5-259dca6fb45f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:38.380220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff533aa4-723d-4f48-a4d5-259dca6fb45f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:38.396921 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-ff533aa4-723d-4f48-a4d5-259dca6fb45f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:38.398114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ff533aa4-723d-4f48-a4d5-259dca6fb45f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:38.428140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 149/310] 10.222.0.47 () {58 vars in 1334 bytes} [Sun Aug 21 01:04:38 2022] PUT /identity/v3/system/users/5eeda698b69748d0ba75340d340cc330/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:04:38.428140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8e130b-e9e1-4b3e-a50d-4fbdfd6a6ed8 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:38.438634 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:38.449061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:38.531507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 162/311] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:38.566138 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8e130b-e9e1-4b3e-a50d-4fbdfd6a6ed8 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:38.606334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8ec430-4f2d-4117-812a-883d2ab62ddf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:38.606933 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8ec430-4f2d-4117-812a-883d2ab62ddf None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:38.607209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8ec430-4f2d-4117-812a-883d2ab62ddf None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:38.607521 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8ec430-4f2d-4117-812a-883d2ab62ddf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:38.609560 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8ec430-4f2d-4117-812a-883d2ab62ddf None admin] RBAC: Adding query filter params (role.id=b5eaad32927541e2a79d4904b5230343, user.id=79047ab97cfc4a83a1a05a84c845c021, scope.system=all) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:38.609560 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8ec430-4f2d-4117-812a-883d2ab62ddf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:38.610066 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8ec430-4f2d-4117-812a-883d2ab62ddf None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:38.618530 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 150/312] 10.222.0.47 () {56 vars in 1308 bytes} [Sun Aug 21 01:04:38 2022] GET /identity/v3/role_assignments?user.id=79047ab97cfc4a83a1a05a84c845c021&scope.system=all&role.id=b5eaad32927541e2a79d4904b5230343 => generated 226 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:38.819626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff533aa4-723d-4f48-a4d5-259dca6fb45f None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:38.930355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff533aa4-723d-4f48-a4d5-259dca6fb45f None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:38.968039 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a175331-7fcd-4d18-907c-9ab04ebd3168 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:38.968714 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a175331-7fcd-4d18-907c-9ab04ebd3168 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:38.968992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a175331-7fcd-4d18-907c-9ab04ebd3168 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:38.969264 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a175331-7fcd-4d18-907c-9ab04ebd3168 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:38.981317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a175331-7fcd-4d18-907c-9ab04ebd3168 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:38.986222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-8a175331-7fcd-4d18-907c-9ab04ebd3168 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:38.986222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:38.986222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:38.986222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:38.986222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:38.986222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:38.986222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:38.986222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:38.986222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:38.986222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:38.986222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:38.986222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:38.986222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:38.986222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:38.986222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:38.986222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:38.987102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:38.987102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:38.987102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:38.987102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:38.987102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:38.987102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:38.987102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:38.987102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:38.987102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:38.987102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:38.987102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:38.987102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:38.987102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:38.987102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:38.987102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:38.987102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:38.987898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:38.987898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:38.987898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:38.987898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:38.987898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:38.987898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:38.987898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:38.987898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:38.987898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:38.987898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:38.987898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:38.987898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:38.987898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 163/313] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:38 2022] GET /identity/v3/roles/reader => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:38.997232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8ec430-4f2d-4117-812a-883d2ab62ddf None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:39.078134 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:39.113663 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8ec430-4f2d-4117-812a-883d2ab62ddf None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:39.152240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ebec81-4b44-41e0-bd47-d5841bb63825 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:39.153221 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ebec81-4b44-41e0-bd47-d5841bb63825 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:39.153676 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ebec81-4b44-41e0-bd47-d5841bb63825 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:39.154119 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ebec81-4b44-41e0-bd47-d5841bb63825 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:39.154954 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ebec81-4b44-41e0-bd47-d5841bb63825 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:39.155429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ebec81-4b44-41e0-bd47-d5841bb63825 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:39.157116 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ebec81-4b44-41e0-bd47-d5841bb63825 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:39.168492 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 151/314] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:38 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:39.174480 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a175331-7fcd-4d18-907c-9ab04ebd3168 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:39.209079 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:39.295439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a175331-7fcd-4d18-907c-9ab04ebd3168 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:39.336485 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c237adb5-e2f0-45d9-a333-d0f6ab56e529 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:39.337458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c237adb5-e2f0-45d9-a333-d0f6ab56e529 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:39.337908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c237adb5-e2f0-45d9-a333-d0f6ab56e529 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:39.338350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c237adb5-e2f0-45d9-a333-d0f6ab56e529 None admin] PATH_INFO: `/v3/users/023bc866d42449b092b1c53d522eb02d` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:39.339769 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c237adb5-e2f0-45d9-a333-d0f6ab56e529 None admin] RBAC: Authorizing `identity:get_user(user_id=023bc866d42449b092b1c53d522eb02d)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:39.341477 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c237adb5-e2f0-45d9-a333-d0f6ab56e529 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:39.343642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 164/315] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:39 2022] GET /identity/v3/users/023bc866d42449b092b1c53d522eb02d => generated 297 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:39.351510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ebec81-4b44-41e0-bd47-d5841bb63825 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:39.462941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ebec81-4b44-41e0-bd47-d5841bb63825 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:39.494182 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:39.499042 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05eea13-3018-4d4c-af91-fe9f10214a3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:39.499652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05eea13-3018-4d4c-af91-fe9f10214a3a None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:39.499928 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05eea13-3018-4d4c-af91-fe9f10214a3a None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:39.500227 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05eea13-3018-4d4c-af91-fe9f10214a3a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:39.580520 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05eea13-3018-4d4c-af91-fe9f10214a3a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:39.582008 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05eea13-3018-4d4c-af91-fe9f10214a3a None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:39.698778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 152/316] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:39.704627 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c237adb5-e2f0-45d9-a333-d0f6ab56e529 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:39.812736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:39.820374 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c237adb5-e2f0-45d9-a333-d0f6ab56e529 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:39.858208 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51df46db-5436-48d5-8805-c9f7fed63420 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:39.859205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51df46db-5436-48d5-8805-c9f7fed63420 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:39.859678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51df46db-5436-48d5-8805-c9f7fed63420 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:39.860167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51df46db-5436-48d5-8805-c9f7fed63420 None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:39.861599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51df46db-5436-48d5-8805-c9f7fed63420 None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:39.863280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51df46db-5436-48d5-8805-c9f7fed63420 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:39.865550 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 165/317] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:39 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:39.873197 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05eea13-3018-4d4c-af91-fe9f10214a3a None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:39.908921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:39.996744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05eea13-3018-4d4c-af91-fe9f10214a3a None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:40.040766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d07098-240b-426f-ad28-29ee05b8d991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:40.041751 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d07098-240b-426f-ad28-29ee05b8d991 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:40.042229 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d07098-240b-426f-ad28-29ee05b8d991 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:40.042760 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d07098-240b-426f-ad28-29ee05b8d991 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:40.043718 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d07098-240b-426f-ad28-29ee05b8d991 None admin] RBAC: Adding query filter params (role.id=b5eaad32927541e2a79d4904b5230343, scope.project.id=ea5d616ff88c481baff44c2b3840d3d4, user.id=023bc866d42449b092b1c53d522eb02d) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:40.044276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d07098-240b-426f-ad28-29ee05b8d991 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:40.045967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d07098-240b-426f-ad28-29ee05b8d991 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:40.053212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 153/318] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:04:39 2022] GET /identity/v3/role_assignments?user.id=023bc866d42449b092b1c53d522eb02d&scope.project.id=ea5d616ff88c481baff44c2b3840d3d4&role.id=b5eaad32927541e2a79d4904b5230343 => generated 608 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:40.129204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c81f35-3d39-490d-893c-ca94050ea876 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:40.129895 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c81f35-3d39-490d-893c-ca94050ea876 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:40.130371 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c81f35-3d39-490d-893c-ca94050ea876 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:40.132029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 166/319] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:40.138713 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ce6fd2-845a-4d2e-8b14-f3c3656609b1 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:40.139058 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ce6fd2-845a-4d2e-8b14-f3c3656609b1 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:40.139277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ce6fd2-845a-4d2e-8b14-f3c3656609b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:40.159370 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-44ce6fd2-845a-4d2e-8b14-f3c3656609b1 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:40.160648 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-44ce6fd2-845a-4d2e-8b14-f3c3656609b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:40.242385 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 154/320] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:40.421070 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfb8583-588e-4615-8fcb-e1b01d3c6ee5 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:40.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfb8583-588e-4615-8fcb-e1b01d3c6ee5 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:40.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfb8583-588e-4615-8fcb-e1b01d3c6ee5 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:40.423012 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 167/321] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:40.428445 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc41c47d-14bc-4e3f-b81a-9d5bf7aab824 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:40.428872 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc41c47d-14bc-4e3f-b81a-9d5bf7aab824 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:40.428969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc41c47d-14bc-4e3f-b81a-9d5bf7aab824 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:40.449956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-fc41c47d-14bc-4e3f-b81a-9d5bf7aab824 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:40.451185 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-fc41c47d-14bc-4e3f-b81a-9d5bf7aab824 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:40.480679 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:40.514743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfb8583-588e-4615-8fcb-e1b01d3c6ee5 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:40.535273 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 155/322] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:40.632241 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfb8583-588e-4615-8fcb-e1b01d3c6ee5 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:40.657540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7bd7f1-7d35-490c-8f85-4b01ce9520b5 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:40.657821 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7bd7f1-7d35-490c-8f85-4b01ce9520b5 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:40.658102 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7bd7f1-7d35-490c-8f85-4b01ce9520b5 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:40.659717 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 156/323] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:40.666511 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0279fc46-baf1-4f75-a8f5-74145c9db113 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:40.667108 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0279fc46-baf1-4f75-a8f5-74145c9db113 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:40.667811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0279fc46-baf1-4f75-a8f5-74145c9db113 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:40.673478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec440d2c-1b7d-46a7-b651-76a353c8d60f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:40.674044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec440d2c-1b7d-46a7-b651-76a353c8d60f None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:40.674312 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec440d2c-1b7d-46a7-b651-76a353c8d60f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:40.674576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec440d2c-1b7d-46a7-b651-76a353c8d60f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:40.675687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec440d2c-1b7d-46a7-b651-76a353c8d60f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:40.677156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec440d2c-1b7d-46a7-b651-76a353c8d60f None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:40.678926 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 168/324] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:04:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:40.684452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec440d2c-1b7d-46a7-b651-76a353c8d60f None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:40.688134 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-0279fc46-baf1-4f75-a8f5-74145c9db113 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:40.689335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-0279fc46-baf1-4f75-a8f5-74145c9db113 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:40.719688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:40.778938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 157/325] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:40.799575 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0279fc46-baf1-4f75-a8f5-74145c9db113 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:40.820054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec440d2c-1b7d-46a7-b651-76a353c8d60f None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:40.864817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6920d718-dc23-405c-8aa8-7e1066ecb58f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:40.865575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6920d718-dc23-405c-8aa8-7e1066ecb58f None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:40.865911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6920d718-dc23-405c-8aa8-7e1066ecb58f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:40.866184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6920d718-dc23-405c-8aa8-7e1066ecb58f None admin] PATH_INFO: `/v3/groups` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:40.867310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6920d718-dc23-405c-8aa8-7e1066ecb58f None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:40.868780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6920d718-dc23-405c-8aa8-7e1066ecb58f None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:40.910412 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:40.927960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0279fc46-baf1-4f75-a8f5-74145c9db113 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:40.959592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 169/326] 10.222.0.47 () {60 vars in 1134 bytes} [Sun Aug 21 01:04:40 2022] POST /identity/v3/groups => generated 226 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:04:40.965258 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c426ab-3629-44b8-9a5c-25fcab277620 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:40.966238 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c426ab-3629-44b8-9a5c-25fcab277620 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:40.966699 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c426ab-3629-44b8-9a5c-25fcab277620 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:40.967148 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c426ab-3629-44b8-9a5c-25fcab277620 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:40.982555 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c426ab-3629-44b8-9a5c-25fcab277620 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:40.988340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-50c426ab-3629-44b8-9a5c-25fcab277620 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:40.988340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:40.988340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:40.988340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:40.988340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:40.988340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:40.988340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:40.988340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:40.988340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:40.988340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:40.988340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:40.988340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:40.988340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:40.988340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:40.988340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:40.988340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:40.989429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:40.989429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:40.989429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:40.989429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:40.989429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:40.989429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:40.989429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:40.989429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:40.989429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:40.989429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:40.989429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:40.989429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:40.989429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:40.989429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:40.989429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:40.989429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:40.990230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:40.990230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:40.990230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:40.990230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:40.990230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:40.990230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:40.990230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:40.990230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:40.990230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:40.990230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:40.990230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:40.990230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:40.996863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 158/327] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:40 2022] GET /identity/v3/roles/member => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:04:40.997543 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6920d718-dc23-405c-8aa8-7e1066ecb58f None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:41.049832 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c426ab-3629-44b8-9a5c-25fcab277620 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:41.131554 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:41.149249 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6920d718-dc23-405c-8aa8-7e1066ecb58f None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:41.195697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c426ab-3629-44b8-9a5c-25fcab277620 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:41.210128 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e52038e-1204-4552-b493-8035a361faea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:41.210786 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e52038e-1204-4552-b493-8035a361faea None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:41.211497 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e52038e-1204-4552-b493-8035a361faea None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:41.211497 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e52038e-1204-4552-b493-8035a361faea None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:41.211990 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e52038e-1204-4552-b493-8035a361faea None admin] RBAC: Adding query filter params (name=member) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:41.213163 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e52038e-1204-4552-b493-8035a361faea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:41.214583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e52038e-1204-4552-b493-8035a361faea None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:41.228633 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 170/328] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:40 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:41.234588 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e52038e-1204-4552-b493-8035a361faea None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:41.247715 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec75b5e2-732c-4e7e-8512-10398190e53c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:41.248697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec75b5e2-732c-4e7e-8512-10398190e53c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:41.249116 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec75b5e2-732c-4e7e-8512-10398190e53c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:41.249531 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec75b5e2-732c-4e7e-8512-10398190e53c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:41.266814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec75b5e2-732c-4e7e-8512-10398190e53c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:41.274017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-ec75b5e2-732c-4e7e-8512-10398190e53c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:41.274017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:41.274017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:41.274017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:41.274017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:41.274017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:41.274017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:41.274017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:41.274017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:41.274017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:41.274017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:41.274017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:41.274017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:41.274017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:41.274017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:41.274017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:41.275842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:41.275842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:41.275842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:41.275842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:41.275842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:41.275842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:41.275842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:41.275842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:41.275842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:41.275842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:41.275842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:41.275842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:41.275842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:41.275842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:41.275842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:41.275842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:41.280989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:41.280989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:41.280989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:41.280989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:41.280989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:41.280989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:41.280989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:41.280989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:41.280989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:41.280989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:41.280989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:41.280989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:41.280989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 159/329] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:41 2022] GET /identity/v3/roles/reader => generated 84 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:04:41.286826 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec75b5e2-732c-4e7e-8512-10398190e53c None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:41.302617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:41.410268 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e52038e-1204-4552-b493-8035a361faea None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:41.463379 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec75b5e2-732c-4e7e-8512-10398190e53c None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:41.484084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1023c4b7-bf1b-432a-a8fb-91b04573048c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:41.485246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1023c4b7-bf1b-432a-a8fb-91b04573048c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:41.485622 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1023c4b7-bf1b-432a-a8fb-91b04573048c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:41.485993 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1023c4b7-bf1b-432a-a8fb-91b04573048c None admin] PATH_INFO: `/v3/users/5eeda698b69748d0ba75340d340cc330` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:41.487561 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1023c4b7-bf1b-432a-a8fb-91b04573048c None admin] RBAC: Authorizing `identity:get_user(user_id=5eeda698b69748d0ba75340d340cc330)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:41.490045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1023c4b7-bf1b-432a-a8fb-91b04573048c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:41.492086 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 171/330] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:41 2022] GET /identity/v3/users/5eeda698b69748d0ba75340d340cc330 => generated 293 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:41.498004 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1023c4b7-bf1b-432a-a8fb-91b04573048c None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:41.511162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9834fb19-7aa6-4315-935b-e5a56b194252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:41.512190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9834fb19-7aa6-4315-935b-e5a56b194252 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:41.512706 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9834fb19-7aa6-4315-935b-e5a56b194252 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:41.513185 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9834fb19-7aa6-4315-935b-e5a56b194252 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:41.514069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9834fb19-7aa6-4315-935b-e5a56b194252 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:41.514568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9834fb19-7aa6-4315-935b-e5a56b194252 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:41.516277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9834fb19-7aa6-4315-935b-e5a56b194252 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:41.526817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 160/331] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:41 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:41.533228 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9834fb19-7aa6-4315-935b-e5a56b194252 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:41.534402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:41.618522 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:41.647690 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1023c4b7-bf1b-432a-a8fb-91b04573048c None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:41.656890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9834fb19-7aa6-4315-935b-e5a56b194252 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:41.689250 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25191ef3-e320-4aec-af7b-dced4a5aecb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:41.690420 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25191ef3-e320-4aec-af7b-dced4a5aecb3 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:41.690915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25191ef3-e320-4aec-af7b-dced4a5aecb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:41.691324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25191ef3-e320-4aec-af7b-dced4a5aecb3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:41.692410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25191ef3-e320-4aec-af7b-dced4a5aecb3 None admin] RBAC: Adding query filter params (role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe, user.id=5eeda698b69748d0ba75340d340cc330, scope.system=all) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:41.692884 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25191ef3-e320-4aec-af7b-dced4a5aecb3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:41.694912 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25191ef3-e320-4aec-af7b-dced4a5aecb3 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:41.697331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93fe8ed-2376-490b-ac6d-a9103041ee34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:41.698086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93fe8ed-2376-490b-ac6d-a9103041ee34 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:41.698460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93fe8ed-2376-490b-ac6d-a9103041ee34 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:41.698848 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93fe8ed-2376-490b-ac6d-a9103041ee34 None admin] PATH_INFO: `/v3/users/79047ab97cfc4a83a1a05a84c845c021` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:41.700260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93fe8ed-2376-490b-ac6d-a9103041ee34 None admin] RBAC: Authorizing `identity:get_user(user_id=79047ab97cfc4a83a1a05a84c845c021)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:41.702809 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93fe8ed-2376-490b-ac6d-a9103041ee34 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:41.706189 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 161/332] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:41 2022] GET /identity/v3/users/79047ab97cfc4a83a1a05a84c845c021 => generated 293 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:41.712374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93fe8ed-2376-490b-ac6d-a9103041ee34 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:41.713929 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 172/333] 10.222.0.47 () {56 vars in 1308 bytes} [Sun Aug 21 01:04:41 2022] GET /identity/v3/role_assignments?user.id=5eeda698b69748d0ba75340d340cc330&scope.system=all&role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 510 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:41.835265 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93fe8ed-2376-490b-ac6d-a9103041ee34 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:41.875166 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096e9299-d927-4785-8110-64b470ce531d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:41.875840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096e9299-d927-4785-8110-64b470ce531d None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:41.876140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096e9299-d927-4785-8110-64b470ce531d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:41.876422 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096e9299-d927-4785-8110-64b470ce531d None admin] PATH_INFO: `/v3/system/users/79047ab97cfc4a83a1a05a84c845c021/roles/b5eaad32927541e2a79d4904b5230343` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:41.877896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096e9299-d927-4785-8110-64b470ce531d None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=79047ab97cfc4a83a1a05a84c845c021, role_id=b5eaad32927541e2a79d4904b5230343)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:41.879298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096e9299-d927-4785-8110-64b470ce531d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:41.882093 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:41.943125 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 162/334] 10.222.0.47 () {58 vars in 1334 bytes} [Sun Aug 21 01:04:41 2022] PUT /identity/v3/system/users/79047ab97cfc4a83a1a05a84c845c021/roles/b5eaad32927541e2a79d4904b5230343 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:04:43.254410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c06e33-c19b-4276-b997-c0efeadbfe2d None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:43.254936 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c06e33-c19b-4276-b997-c0efeadbfe2d None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:43.254936 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c06e33-c19b-4276-b997-c0efeadbfe2d None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:43.256727 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 173/335] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:04:43.267188 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915c3379-92bb-4700-b298-4bde6e43162f None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:43.267715 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915c3379-92bb-4700-b298-4bde6e43162f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:43.267715 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915c3379-92bb-4700-b298-4bde6e43162f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:43.293359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-915c3379-92bb-4700-b298-4bde6e43162f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:43.294578 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-915c3379-92bb-4700-b298-4bde6e43162f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:43.399993 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 163/336] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:04:43.584036 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec04e01-d08e-47c3-abcf-6036d86deeea None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:43.584405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec04e01-d08e-47c3-abcf-6036d86deeea None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:43.584714 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec04e01-d08e-47c3-abcf-6036d86deeea None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:43.586314 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 174/337] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:43.591829 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed7d677-2be8-49bd-9a22-29533c8b7334 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:43.592130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed7d677-2be8-49bd-9a22-29533c8b7334 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:43.592404 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed7d677-2be8-49bd-9a22-29533c8b7334 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:43.617967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-aed7d677-2be8-49bd-9a22-29533c8b7334 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:43.619237 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-aed7d677-2be8-49bd-9a22-29533c8b7334 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:43.711009 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 164/338] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:44.071100 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec04e01-d08e-47c3-abcf-6036d86deeea None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:44.184788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:44.211608 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec04e01-d08e-47c3-abcf-6036d86deeea None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:44.261696 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc93e57-be16-4171-ade2-12a081a59111 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:44.262680 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc93e57-be16-4171-ade2-12a081a59111 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:44.263123 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc93e57-be16-4171-ade2-12a081a59111 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:44.263550 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc93e57-be16-4171-ade2-12a081a59111 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:44.277094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc93e57-be16-4171-ade2-12a081a59111 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:44.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-0fc93e57-be16-4171-ade2-12a081a59111 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:44.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:44.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:44.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:44.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:44.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:44.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:44.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:44.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:44.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:44.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:44.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:44.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:44.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:44.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:44.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:44.284132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:44.284132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:44.284132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:44.284132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:44.284132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:44.284132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:44.284132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:44.284132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:44.284132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:44.284132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:44.284132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:44.284132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:44.284132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:44.284132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:44.284132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:44.284132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:44.285049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:44.285049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:44.285049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:44.285049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:44.285049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:44.285049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:44.285049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:44.285049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:44.285049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:44.285049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:44.285049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:44.285049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:44.285049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 175/339] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:44.291676 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed7d677-2be8-49bd-9a22-29533c8b7334 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:44.309940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc93e57-be16-4171-ade2-12a081a59111 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:44.414703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:44.432878 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc93e57-be16-4171-ade2-12a081a59111 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:44.474396 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8ec069-9d06-46d5-8880-a1090923512e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:44.475275 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8ec069-9d06-46d5-8880-a1090923512e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:44.475687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8ec069-9d06-46d5-8880-a1090923512e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:44.476194 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8ec069-9d06-46d5-8880-a1090923512e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:44.477760 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed7d677-2be8-49bd-9a22-29533c8b7334 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:44.487062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:44.490962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8ec069-9d06-46d5-8880-a1090923512e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:44.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-7e8ec069-9d06-46d5-8880-a1090923512e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:44.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:44.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:44.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:44.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:44.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:44.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:44.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:44.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:44.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:44.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:44.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:44.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:44.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:44.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:44.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:44.500150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:44.500150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:44.500150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:44.500150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:44.500150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:44.500150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:44.500150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:44.500150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:44.500150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:44.500150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:44.500150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:44.500150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:44.500150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:44.500150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:44.500150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:44.500150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:44.500942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:44.500942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:44.500942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:44.500942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:44.500942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:44.500942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:44.500942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:44.500942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:44.500942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:44.500942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:44.500942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:44.500942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:44.500942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 176/340] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:04:44 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:04:44.500942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe40c407-d1d2-4f8b-81bb-bea1e9a5dce5 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:44.501649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe40c407-d1d2-4f8b-81bb-bea1e9a5dce5 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:44.501649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe40c407-d1d2-4f8b-81bb-bea1e9a5dce5 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:44.501649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 177/341] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:44.506494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe40c407-d1d2-4f8b-81bb-bea1e9a5dce5 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:44.528371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c259ca-b1f0-4afd-9b42-6d73bc187a74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:44.529438 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c259ca-b1f0-4afd-9b42-6d73bc187a74 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:44.529911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c259ca-b1f0-4afd-9b42-6d73bc187a74 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:44.530400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c259ca-b1f0-4afd-9b42-6d73bc187a74 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:44.531260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c259ca-b1f0-4afd-9b42-6d73bc187a74 None admin] RBAC: Adding query filter params (name=member) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:44.531779 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c259ca-b1f0-4afd-9b42-6d73bc187a74 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:44.533501 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c259ca-b1f0-4afd-9b42-6d73bc187a74 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:44.545115 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 165/342] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:44.546716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968d5aeb-bdb6-4d9b-920e-cbfe0329f7b9 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:44.547244 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968d5aeb-bdb6-4d9b-920e-cbfe0329f7b9 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:44.547719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968d5aeb-bdb6-4d9b-920e-cbfe0329f7b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:44.568030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-968d5aeb-bdb6-4d9b-920e-cbfe0329f7b9 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:44.570486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-968d5aeb-bdb6-4d9b-920e-cbfe0329f7b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:44.622937 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe40c407-d1d2-4f8b-81bb-bea1e9a5dce5 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:44.651278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 166/343] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:44.652528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968d5aeb-bdb6-4d9b-920e-cbfe0329f7b9 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:44.666441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ffaf77-693a-4fe9-818e-6198eac561a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:44.667080 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ffaf77-693a-4fe9-818e-6198eac561a1 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:44.667624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ffaf77-693a-4fe9-818e-6198eac561a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:44.667730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ffaf77-693a-4fe9-818e-6198eac561a1 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:44.668456 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ffaf77-693a-4fe9-818e-6198eac561a1 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:44.668910 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ffaf77-693a-4fe9-818e-6198eac561a1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:44.670512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ffaf77-693a-4fe9-818e-6198eac561a1 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:44.681714 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 178/344] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:04:44 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:44.689980 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ffaf77-693a-4fe9-818e-6198eac561a1 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:44.750332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:44.784516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:44.831659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ffaf77-693a-4fe9-818e-6198eac561a1 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:44.877358 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968d5aeb-bdb6-4d9b-920e-cbfe0329f7b9 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:44.895502 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b256869-f210-4103-992f-8c33b5aa3b55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:44.896395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b256869-f210-4103-992f-8c33b5aa3b55 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:44.896834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b256869-f210-4103-992f-8c33b5aa3b55 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:44.897244 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b256869-f210-4103-992f-8c33b5aa3b55 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:44.953195 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3335214a-af09-492c-8350-c8446b9c8a33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:44.954122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3335214a-af09-492c-8350-c8446b9c8a33 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:44.954535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3335214a-af09-492c-8350-c8446b9c8a33 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:44.954979 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3335214a-af09-492c-8350-c8446b9c8a33 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:45.059951 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b256869-f210-4103-992f-8c33b5aa3b55 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:45.061685 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b256869-f210-4103-992f-8c33b5aa3b55 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:45.127807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3335214a-af09-492c-8350-c8446b9c8a33 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:45.129529 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3335214a-af09-492c-8350-c8446b9c8a33 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:45.213492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:45.255931 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:45.292257 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 179/345] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 603 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:45.293145 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b256869-f210-4103-992f-8c33b5aa3b55 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:45.350369 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 167/346] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 698 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:45.353531 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc553084-4a4b-46c4-8ef0-6b30903528c3 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:45.354034 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc553084-4a4b-46c4-8ef0-6b30903528c3 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:45.354495 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc553084-4a4b-46c4-8ef0-6b30903528c3 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:45.358766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 168/347] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:45 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:45.363850 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc553084-4a4b-46c4-8ef0-6b30903528c3 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:45.445646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b256869-f210-4103-992f-8c33b5aa3b55 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:45.508931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6de6ab6-27a9-4469-9b35-c0d875f50507 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:45.510127 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6de6ab6-27a9-4469-9b35-c0d875f50507 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:45.510749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6de6ab6-27a9-4469-9b35-c0d875f50507 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:45.511025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6de6ab6-27a9-4469-9b35-c0d875f50507 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:45.520580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc553084-4a4b-46c4-8ef0-6b30903528c3 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:45.527003 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6de6ab6-27a9-4469-9b35-c0d875f50507 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:45.532779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-b6de6ab6-27a9-4469-9b35-c0d875f50507 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:45.532779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:45.532779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:45.532779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:45.532779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:45.532779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:45.532779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:45.532779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:45.532779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:45.532779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:45.532779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:45.532779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:45.532779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:45.532779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:45.532779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:45.532779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:45.533822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:45.533822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:45.533822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:45.533822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:45.533822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:45.533822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:45.533822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:45.533822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:45.533822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:45.533822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:45.533822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:45.533822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:45.533822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:45.533822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:45.533822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:45.533822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:45.534731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:45.534731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:45.534731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:45.534731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:45.534731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:45.534731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:45.534731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:45.534731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:45.534731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:45.534731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:45.534731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:45.534731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:45.536059 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 180/348] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:45.536851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6de6ab6-27a9-4469-9b35-c0d875f50507 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:45.574561 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7fb980b-bf71-4995-9f1e-6e05ff7db34b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:45.575732 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:45.576694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7fb980b-bf71-4995-9f1e-6e05ff7db34b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:45.577232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7fb980b-bf71-4995-9f1e-6e05ff7db34b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:45.577703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7fb980b-bf71-4995-9f1e-6e05ff7db34b None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:45.579258 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7fb980b-bf71-4995-9f1e-6e05ff7db34b None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:45.581071 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7fb980b-bf71-4995-9f1e-6e05ff7db34b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:45.583234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 169/349] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:45 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:45.584971 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c7f94d-a306-4e6c-8568-b2ca03ebf452 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:45.586011 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c7f94d-a306-4e6c-8568-b2ca03ebf452 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:45.586447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c7f94d-a306-4e6c-8568-b2ca03ebf452 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:45.615517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-44c7f94d-a306-4e6c-8568-b2ca03ebf452 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:45.618395 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-44c7f94d-a306-4e6c-8568-b2ca03ebf452 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:45.646962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:45.674182 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6de6ab6-27a9-4469-9b35-c0d875f50507 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:45.718316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd99fe1-1193-445f-a675-aecac958c721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:45.719388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd99fe1-1193-445f-a675-aecac958c721 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:45.719840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd99fe1-1193-445f-a675-aecac958c721 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:45.720341 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd99fe1-1193-445f-a675-aecac958c721 None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:45.721763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd99fe1-1193-445f-a675-aecac958c721 None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:45.723503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd99fe1-1193-445f-a675-aecac958c721 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:45.724377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 170/350] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:45.725484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c7f94d-a306-4e6c-8568-b2ca03ebf452 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:45.726528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 181/351] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:45 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:45.727550 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd99fe1-1193-445f-a675-aecac958c721 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:45.870689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c7f94d-a306-4e6c-8568-b2ca03ebf452 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:45.883034 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:45.899587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd99fe1-1193-445f-a675-aecac958c721 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:45.918204 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac390650-ea5d-488e-82c1-1acab192bdea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:45.919003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac390650-ea5d-488e-82c1-1acab192bdea None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:45.920214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac390650-ea5d-488e-82c1-1acab192bdea None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:45.920214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac390650-ea5d-488e-82c1-1acab192bdea None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:45.920407 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac390650-ea5d-488e-82c1-1acab192bdea None admin] RBAC: Adding query filter params (name=member) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:45.920896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac390650-ea5d-488e-82c1-1acab192bdea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:45.926008 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac390650-ea5d-488e-82c1-1acab192bdea None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:45.933046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 171/352] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:45.933901 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac390650-ea5d-488e-82c1-1acab192bdea None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:45.950622 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:45.960248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20327e22-1097-4517-bf11-c98389cb23a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:45.962537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20327e22-1097-4517-bf11-c98389cb23a0 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:45.962537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20327e22-1097-4517-bf11-c98389cb23a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:45.962537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20327e22-1097-4517-bf11-c98389cb23a0 None admin] PATH_INFO: `/v3/groups/9ba79133e99b4e91b58e94e32f101d4d` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:45.973137 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20327e22-1097-4517-bf11-c98389cb23a0 None admin] RBAC: Authorizing `identity:get_group(group_id=9ba79133e99b4e91b58e94e32f101d4d)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:45.974943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20327e22-1097-4517-bf11-c98389cb23a0 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:45.977739 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 182/353] 10.222.0.47 () {56 vars in 1180 bytes} [Sun Aug 21 01:04:45 2022] GET /identity/v3/groups/9ba79133e99b4e91b58e94e32f101d4d => generated 226 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:45.979329 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b131010-b944-4651-9633-0c6d2d01c67a None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:45.979759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b131010-b944-4651-9633-0c6d2d01c67a None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:45.980227 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b131010-b944-4651-9633-0c6d2d01c67a None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:45.982065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 183/354] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:45.983358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6b6e38-d601-4f10-898a-216a7786bec5 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:45.984042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6b6e38-d601-4f10-898a-216a7786bec5 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:45.984516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6b6e38-d601-4f10-898a-216a7786bec5 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:45.986083 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 184/355] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:45.986988 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6b6e38-d601-4f10-898a-216a7786bec5 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:46.072341 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac390650-ea5d-488e-82c1-1acab192bdea None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:46.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45e6517-da99-4f63-8a42-68044ce32538 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:46.118869 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45e6517-da99-4f63-8a42-68044ce32538 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:46.119133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45e6517-da99-4f63-8a42-68044ce32538 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:46.119399 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45e6517-da99-4f63-8a42-68044ce32538 None admin] PATH_INFO: `/v3/groups/9ba79133e99b4e91b58e94e32f101d4d` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:46.122960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45e6517-da99-4f63-8a42-68044ce32538 None admin] RBAC: Authorizing `identity:get_group(group_id=9ba79133e99b4e91b58e94e32f101d4d)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:46.124461 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45e6517-da99-4f63-8a42-68044ce32538 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:46.126148 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 172/356] 10.222.0.47 () {56 vars in 1180 bytes} [Sun Aug 21 01:04:45 2022] GET /identity/v3/groups/9ba79133e99b4e91b58e94e32f101d4d => generated 226 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:46.126969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45e6517-da99-4f63-8a42-68044ce32538 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:46.129739 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6b6e38-d601-4f10-898a-216a7786bec5 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:46.175583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e64b69-af98-4a2e-a616-e540f0484fa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:46.176240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e64b69-af98-4a2e-a616-e540f0484fa1 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:46.176508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e64b69-af98-4a2e-a616-e540f0484fa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:46.176785 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e64b69-af98-4a2e-a616-e540f0484fa1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:46.215619 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:46.240358 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:46.273876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e64b69-af98-4a2e-a616-e540f0484fa1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:46.276808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e64b69-af98-4a2e-a616-e540f0484fa1 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:46.281932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45e6517-da99-4f63-8a42-68044ce32538 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:46.334519 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76728ac-f301-46af-b77c-d59c86e82108 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:46.336092 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76728ac-f301-46af-b77c-d59c86e82108 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:46.336573 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76728ac-f301-46af-b77c-d59c86e82108 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:46.337040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76728ac-f301-46af-b77c-d59c86e82108 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:46.337936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76728ac-f301-46af-b77c-d59c86e82108 None admin] RBAC: Adding query filter params (group.id=9ba79133e99b4e91b58e94e32f101d4d, role.id=fbc5961470b74a85bde6be2171f5f08f, scope.project.id=63ed5323666040089b15189fade45bf4) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:46.340192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76728ac-f301-46af-b77c-d59c86e82108 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:46.340192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76728ac-f301-46af-b77c-d59c86e82108 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:46.348969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 173/357] 10.222.0.47 () {56 vars in 1376 bytes} [Sun Aug 21 01:04:46 2022] GET /identity/v3/role_assignments?group.id=9ba79133e99b4e91b58e94e32f101d4d&scope.project.id=63ed5323666040089b15189fade45bf4&role.id=fbc5961470b74a85bde6be2171f5f08f => generated 260 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:46.358920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6edba6b-ef44-474c-95bb-12f6189f784f None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:46.359537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6edba6b-ef44-474c-95bb-12f6189f784f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:46.359537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6edba6b-ef44-474c-95bb-12f6189f784f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:46.381064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-d6edba6b-ef44-474c-95bb-12f6189f784f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:46.382927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d6edba6b-ef44-474c-95bb-12f6189f784f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:46.436160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 185/358] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 447 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:46.436160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218654ad-2fb1-4bd9-92dd-0c26b4fd7e32 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:46.436160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218654ad-2fb1-4bd9-92dd-0c26b4fd7e32 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:46.436160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218654ad-2fb1-4bd9-92dd-0c26b4fd7e32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:46.459150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-218654ad-2fb1-4bd9-92dd-0c26b4fd7e32 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:46.461081 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-218654ad-2fb1-4bd9-92dd-0c26b4fd7e32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:46.491435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 174/359] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:46.492319 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6edba6b-ef44-474c-95bb-12f6189f784f None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:46.560537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:46.582994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 186/360] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:46.584153 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218654ad-2fb1-4bd9-92dd-0c26b4fd7e32 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:46.619055 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:46.662731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6edba6b-ef44-474c-95bb-12f6189f784f None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:46.707614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bafe2c6-cceb-489d-b8e1-82328b8f78f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:46.708327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bafe2c6-cceb-489d-b8e1-82328b8f78f4 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:46.708611 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bafe2c6-cceb-489d-b8e1-82328b8f78f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:46.709100 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bafe2c6-cceb-489d-b8e1-82328b8f78f4 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:46.725033 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bafe2c6-cceb-489d-b8e1-82328b8f78f4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:46.731782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-6bafe2c6-cceb-489d-b8e1-82328b8f78f4 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:46.731782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:46.731782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:46.731782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:46.731782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:46.731782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:46.731782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:46.731782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:46.731782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:46.731782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:46.731782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:46.731782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:46.731782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:46.731782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:46.731782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:46.731782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:46.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:46.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:46.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:46.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:46.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:46.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:46.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:46.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:46.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:46.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:46.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:46.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:46.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:46.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:46.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:46.732770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:46.733641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:46.733641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:46.733641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:46.733641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:46.733641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:46.733641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:46.733641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:46.733641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:46.733641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:46.733641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:46.733641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 21 01:04:46.733641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:46.733641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 175/361] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:46 2022] GET /identity/v3/roles/reader => generated 84 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:46.734322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bafe2c6-cceb-489d-b8e1-82328b8f78f4 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:46.795318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218654ad-2fb1-4bd9-92dd-0c26b4fd7e32 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:46.856299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f496bbae-9c24-481d-b658-056a5b6664a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:46.857359 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f496bbae-9c24-481d-b658-056a5b6664a3 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:46.857635 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f496bbae-9c24-481d-b658-056a5b6664a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:46.857953 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f496bbae-9c24-481d-b658-056a5b6664a3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:46.858809 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f496bbae-9c24-481d-b658-056a5b6664a3 None admin] RBAC: Adding query filter params (group.id=9ba79133e99b4e91b58e94e32f101d4d, role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe, scope.project.id=63ed5323666040089b15189fade45bf4) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:46.859203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f496bbae-9c24-481d-b658-056a5b6664a3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:46.860722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f496bbae-9c24-481d-b658-056a5b6664a3 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:46.869463 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 187/362] 10.222.0.47 () {56 vars in 1376 bytes} [Sun Aug 21 01:04:46 2022] GET /identity/v3/role_assignments?group.id=9ba79133e99b4e91b58e94e32f101d4d&scope.project.id=63ed5323666040089b15189fade45bf4&role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 260 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:46.870885 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f496bbae-9c24-481d-b658-056a5b6664a3 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:46.872878 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bafe2c6-cceb-489d-b8e1-82328b8f78f4 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:46.931188 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37df5f96-8e5b-4455-9214-25251e10ebed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:46.932264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37df5f96-8e5b-4455-9214-25251e10ebed None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:46.932722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37df5f96-8e5b-4455-9214-25251e10ebed None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:46.933159 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37df5f96-8e5b-4455-9214-25251e10ebed None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:46.935551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37df5f96-8e5b-4455-9214-25251e10ebed None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:46.937094 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37df5f96-8e5b-4455-9214-25251e10ebed None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:46.940162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 176/363] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:46 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:46.941067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37df5f96-8e5b-4455-9214-25251e10ebed None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:46.944531 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:46.949266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:47.027789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f496bbae-9c24-481d-b658-056a5b6664a3 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:47.074405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72976203-fe3a-4a93-aaeb-74f39ca2db81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:47.074949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72976203-fe3a-4a93-aaeb-74f39ca2db81 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:47.075221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72976203-fe3a-4a93-aaeb-74f39ca2db81 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:47.075521 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72976203-fe3a-4a93-aaeb-74f39ca2db81 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:47.076563 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72976203-fe3a-4a93-aaeb-74f39ca2db81 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:47.076863 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72976203-fe3a-4a93-aaeb-74f39ca2db81 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:47.079347 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72976203-fe3a-4a93-aaeb-74f39ca2db81 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:47.096221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 188/364] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:46 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:47.097273 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72976203-fe3a-4a93-aaeb-74f39ca2db81 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:47.108699 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37df5f96-8e5b-4455-9214-25251e10ebed None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:47.168312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e462ded8-3244-4b97-9805-ac2c57ac4858 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:47.170011 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e462ded8-3244-4b97-9805-ac2c57ac4858 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:47.171366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e462ded8-3244-4b97-9805-ac2c57ac4858 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:47.172509 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e462ded8-3244-4b97-9805-ac2c57ac4858 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:47.197448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e462ded8-3244-4b97-9805-ac2c57ac4858 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:47.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-e462ded8-3244-4b97-9805-ac2c57ac4858 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:47.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:47.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:47.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:47.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:47.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:47.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:47.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:47.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:47.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:47.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:47.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:47.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:47.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:47.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:47.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:47.204476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:47.204476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:47.204476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:47.204476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:47.204476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:47.204476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:47.204476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:47.204476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:47.204476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:47.204476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:47.204476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:47.204476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:47.204476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:47.204476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:47.204476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:47.204476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:47.205293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:47.205293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:47.205293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:47.205293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:47.205293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:47.205293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:47.205293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:47.205293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:47.205293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:47.205293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:47.205293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:47.205293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:47.205293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 177/365] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:04:46 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:47.205958 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e462ded8-3244-4b97-9805-ac2c57ac4858 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:47.262252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72976203-fe3a-4a93-aaeb-74f39ca2db81 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:47.278665 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:47.298474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:47.324054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7314f1-5642-4921-8958-d133aacf68bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:47.325190 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7314f1-5642-4921-8958-d133aacf68bc None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:47.325625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7314f1-5642-4921-8958-d133aacf68bc None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:47.326063 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7314f1-5642-4921-8958-d133aacf68bc None admin] PATH_INFO: `/v3/groups/9ba79133e99b4e91b58e94e32f101d4d` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:47.327417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7314f1-5642-4921-8958-d133aacf68bc None admin] RBAC: Authorizing `identity:get_group(group_id=9ba79133e99b4e91b58e94e32f101d4d)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:47.329249 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7314f1-5642-4921-8958-d133aacf68bc None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:47.331717 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 189/366] 10.222.0.47 () {56 vars in 1180 bytes} [Sun Aug 21 01:04:47 2022] GET /identity/v3/groups/9ba79133e99b4e91b58e94e32f101d4d => generated 226 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:47.337044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7314f1-5642-4921-8958-d133aacf68bc None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:47.365482 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e462ded8-3244-4b97-9805-ac2c57ac4858 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:47.411355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4d95fe-a6b6-4b9a-95f7-44b510ef4eba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:47.412059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4d95fe-a6b6-4b9a-95f7-44b510ef4eba None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:47.412552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4d95fe-a6b6-4b9a-95f7-44b510ef4eba None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:47.412792 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4d95fe-a6b6-4b9a-95f7-44b510ef4eba None admin] PATH_INFO: `/v3/users/79047ab97cfc4a83a1a05a84c845c021` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:47.414562 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4d95fe-a6b6-4b9a-95f7-44b510ef4eba None admin] RBAC: Authorizing `identity:get_user(user_id=79047ab97cfc4a83a1a05a84c845c021)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:47.416035 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4d95fe-a6b6-4b9a-95f7-44b510ef4eba None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:47.418202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 178/367] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:04:47 2022] GET /identity/v3/users/79047ab97cfc4a83a1a05a84c845c021 => generated 293 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:47.419122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4d95fe-a6b6-4b9a-95f7-44b510ef4eba None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:47.493603 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7314f1-5642-4921-8958-d133aacf68bc None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:47.535304 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87f425d-bb07-4cf9-adc9-1f07e4e2086f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:47.535671 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4d95fe-a6b6-4b9a-95f7-44b510ef4eba None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:47.536602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87f425d-bb07-4cf9-adc9-1f07e4e2086f None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:47.537219 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87f425d-bb07-4cf9-adc9-1f07e4e2086f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:47.537715 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87f425d-bb07-4cf9-adc9-1f07e4e2086f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:47.550522 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87f425d-bb07-4cf9-adc9-1f07e4e2086f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:47.558807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-d87f425d-bb07-4cf9-adc9-1f07e4e2086f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:47.558807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:47.558807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:47.558807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:47.558807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:47.558807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:47.558807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:47.558807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:47.558807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:47.558807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:47.558807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:47.558807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:47.558807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:47.558807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:47.558807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:47.558807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:47.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:47.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:47.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:47.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:47.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:47.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:47.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:47.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:47.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:47.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:47.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:47.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:47.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:47.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:47.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:47.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:47.562064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:47.562064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:47.562064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:47.562064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:47.562064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:47.562064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:47.562064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:47.562064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:47.562064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:47.562064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:47.562064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:47.562064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:47.565131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 190/368] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:04:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:47.566482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87f425d-bb07-4cf9-adc9-1f07e4e2086f None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:47.574613 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:47.590153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a06e0f0-fe6d-46f8-93b4-55615c871cdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:47.590673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a06e0f0-fe6d-46f8-93b4-55615c871cdf None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:47.590974 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a06e0f0-fe6d-46f8-93b4-55615c871cdf None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:47.591253 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a06e0f0-fe6d-46f8-93b4-55615c871cdf None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:47.591900 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a06e0f0-fe6d-46f8-93b4-55615c871cdf None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:47.592232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a06e0f0-fe6d-46f8-93b4-55615c871cdf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:47.593759 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a06e0f0-fe6d-46f8-93b4-55615c871cdf None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:47.607670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 179/369] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:04:47 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:47.609020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a06e0f0-fe6d-46f8-93b4-55615c871cdf None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:47.638200 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:47.735410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87f425d-bb07-4cf9-adc9-1f07e4e2086f None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:47.747552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a06e0f0-fe6d-46f8-93b4-55615c871cdf None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:47.788278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ffb00b-88ab-43d7-a59e-b8fced4a4c29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:47.789813 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ffb00b-88ab-43d7-a59e-b8fced4a4c29 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:47.790335 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ffb00b-88ab-43d7-a59e-b8fced4a4c29 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:47.790855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ffb00b-88ab-43d7-a59e-b8fced4a4c29 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:47.792014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ffb00b-88ab-43d7-a59e-b8fced4a4c29 None admin] RBAC: Adding query filter params (group.id=9ba79133e99b4e91b58e94e32f101d4d, role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe, scope.project.id=ea5d616ff88c481baff44c2b3840d3d4) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:47.792457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ffb00b-88ab-43d7-a59e-b8fced4a4c29 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:47.793759 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70f0273-819d-445e-b183-fe9bf5ed8997 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:47.794125 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ffb00b-88ab-43d7-a59e-b8fced4a4c29 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:47.794699 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70f0273-819d-445e-b183-fe9bf5ed8997 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:47.795119 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70f0273-819d-445e-b183-fe9bf5ed8997 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:47.795525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70f0273-819d-445e-b183-fe9bf5ed8997 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:47.796507 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70f0273-819d-445e-b183-fe9bf5ed8997 None admin] RBAC: Adding query filter params (role.id=b5eaad32927541e2a79d4904b5230343, user.id=79047ab97cfc4a83a1a05a84c845c021, scope.system=all) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:47.796813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70f0273-819d-445e-b183-fe9bf5ed8997 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:47.798287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70f0273-819d-445e-b183-fe9bf5ed8997 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:47.802165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 191/370] 10.222.0.47 () {56 vars in 1376 bytes} [Sun Aug 21 01:04:47 2022] GET /identity/v3/role_assignments?group.id=9ba79133e99b4e91b58e94e32f101d4d&scope.project.id=ea5d616ff88c481baff44c2b3840d3d4&role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 260 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:47.803561 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ffb00b-88ab-43d7-a59e-b8fced4a4c29 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:47.826172 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 180/371] 10.222.0.47 () {56 vars in 1308 bytes} [Sun Aug 21 01:04:47 2022] GET /identity/v3/role_assignments?user.id=79047ab97cfc4a83a1a05a84c845c021&scope.system=all&role.id=b5eaad32927541e2a79d4904b5230343 => generated 510 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:47.827050 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70f0273-819d-445e-b183-fe9bf5ed8997 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:47.947488 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:48.004169 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ffb00b-88ab-43d7-a59e-b8fced4a4c29 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:48.006089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:48.058753 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af141f1d-38a0-4933-ad45-8b5e05548388 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:48.059381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af141f1d-38a0-4933-ad45-8b5e05548388 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:48.059771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af141f1d-38a0-4933-ad45-8b5e05548388 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:48.060051 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af141f1d-38a0-4933-ad45-8b5e05548388 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:48.060767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af141f1d-38a0-4933-ad45-8b5e05548388 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:48.061071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af141f1d-38a0-4933-ad45-8b5e05548388 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:48.062528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af141f1d-38a0-4933-ad45-8b5e05548388 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:48.080068 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 192/372] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:48.083182 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70f0273-819d-445e-b183-fe9bf5ed8997 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:48.087366 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af141f1d-38a0-4933-ad45-8b5e05548388 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:48.168976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c469d7ce-38dc-4c3b-b5a2-10efac4830e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:48.169611 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c469d7ce-38dc-4c3b-b5a2-10efac4830e2 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:48.169883 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c469d7ce-38dc-4c3b-b5a2-10efac4830e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:48.170154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c469d7ce-38dc-4c3b-b5a2-10efac4830e2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:48.282368 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af141f1d-38a0-4933-ad45-8b5e05548388 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:48.301172 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c469d7ce-38dc-4c3b-b5a2-10efac4830e2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:48.302804 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c469d7ce-38dc-4c3b-b5a2-10efac4830e2 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:48.353670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad879f1-c816-45ee-bf2b-eb49bd0bbca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:48.354452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad879f1-c816-45ee-bf2b-eb49bd0bbca5 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:48.354731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad879f1-c816-45ee-bf2b-eb49bd0bbca5 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:48.355001 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad879f1-c816-45ee-bf2b-eb49bd0bbca5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:48.399991 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:48.407028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:48.452067 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad879f1-c816-45ee-bf2b-eb49bd0bbca5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:48.453573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad879f1-c816-45ee-bf2b-eb49bd0bbca5 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:48.500650 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 181/373] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 674 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:48.506945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c469d7ce-38dc-4c3b-b5a2-10efac4830e2 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:48.564283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 193/374] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 477 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:48.571113 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad879f1-c816-45ee-bf2b-eb49bd0bbca5 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:48.653467 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:48.693672 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad879f1-c816-45ee-bf2b-eb49bd0bbca5 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:48.729831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a72e67-0c39-4262-911a-b5478b09c0ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:48.730476 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a72e67-0c39-4262-911a-b5478b09c0ff None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:48.730751 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a72e67-0c39-4262-911a-b5478b09c0ff None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:48.731020 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a72e67-0c39-4262-911a-b5478b09c0ff None admin] PATH_INFO: `/v3/projects/ada79d28652f4a3aa0a174133f24108f` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:48.732312 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a72e67-0c39-4262-911a-b5478b09c0ff None admin] RBAC: Authorizing `identity:get_project(project_id=ada79d28652f4a3aa0a174133f24108f)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:48.733790 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a72e67-0c39-4262-911a-b5478b09c0ff None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:48.735634 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 194/375] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:48 2022] GET /identity/v3/projects/ada79d28652f4a3aa0a174133f24108f => generated 344 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:48.737001 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93398683-7edd-459d-9ae3-11d7b51fa24e None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:48.737158 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93398683-7edd-459d-9ae3-11d7b51fa24e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:48.737525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93398683-7edd-459d-9ae3-11d7b51fa24e None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:48.738872 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 195/376] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:48 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:48.743270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93398683-7edd-459d-9ae3-11d7b51fa24e None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:48.746432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c469d7ce-38dc-4c3b-b5a2-10efac4830e2 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:48.797409 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f52859-e449-4a2b-bba9-69930c218794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:48.798499 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f52859-e449-4a2b-bba9-69930c218794 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:48.799144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f52859-e449-4a2b-bba9-69930c218794 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:48.799619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f52859-e449-4a2b-bba9-69930c218794 None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:48.801071 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f52859-e449-4a2b-bba9-69930c218794 None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:48.802975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f52859-e449-4a2b-bba9-69930c218794 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:48.805747 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 182/377] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:48 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:48.807237 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b99d70b-8054-48ba-abb6-c9ab297824fa None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:48.807701 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b99d70b-8054-48ba-abb6-c9ab297824fa None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:48.808025 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b99d70b-8054-48ba-abb6-c9ab297824fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:48.828267 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-7b99d70b-8054-48ba-abb6-c9ab297824fa None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:48.829446 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7b99d70b-8054-48ba-abb6-c9ab297824fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:48.853580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:48.885482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93398683-7edd-459d-9ae3-11d7b51fa24e None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:48.935610 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:48.942196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 183/378] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:48.943034 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b99d70b-8054-48ba-abb6-c9ab297824fa None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:48.943155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd17ec89-85ae-4392-bbea-566a253214d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:48.944841 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd17ec89-85ae-4392-bbea-566a253214d4 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:48.945804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd17ec89-85ae-4392-bbea-566a253214d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:48.946297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd17ec89-85ae-4392-bbea-566a253214d4 None admin] PATH_INFO: `/v3/groups/22c4cfab6749435b9afdacfff854dd85` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:48.954659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd17ec89-85ae-4392-bbea-566a253214d4 None admin] RBAC: Authorizing `identity:get_group(group_id=22c4cfab6749435b9afdacfff854dd85)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:48.956899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd17ec89-85ae-4392-bbea-566a253214d4 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:48.960810 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 196/379] 10.222.0.47 () {56 vars in 1180 bytes} [Sun Aug 21 01:04:48 2022] GET /identity/v3/groups/22c4cfab6749435b9afdacfff854dd85 => generated 229 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:48.966992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd17ec89-85ae-4392-bbea-566a253214d4 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:49.094864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd17ec89-85ae-4392-bbea-566a253214d4 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:49.136378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef749b59-50a4-4d81-b0e0-5cca2fb8096a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:49.137009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef749b59-50a4-4d81-b0e0-5cca2fb8096a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:49.137290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef749b59-50a4-4d81-b0e0-5cca2fb8096a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:49.137565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef749b59-50a4-4d81-b0e0-5cca2fb8096a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:49.138371 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef749b59-50a4-4d81-b0e0-5cca2fb8096a None admin] RBAC: Adding query filter params (group.id=22c4cfab6749435b9afdacfff854dd85, role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=ada79d28652f4a3aa0a174133f24108f) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:49.138683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef749b59-50a4-4d81-b0e0-5cca2fb8096a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:49.140156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef749b59-50a4-4d81-b0e0-5cca2fb8096a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:49.147968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 197/380] 10.222.0.47 () {56 vars in 1376 bytes} [Sun Aug 21 01:04:48 2022] GET /identity/v3/role_assignments?group.id=22c4cfab6749435b9afdacfff854dd85&scope.project.id=ada79d28652f4a3aa0a174133f24108f&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 260 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:49.236777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b99d70b-8054-48ba-abb6-c9ab297824fa None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:49.310674 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55e7c0e-8f95-4fbf-8f64-d698998d5f17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:49.311944 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55e7c0e-8f95-4fbf-8f64-d698998d5f17 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:49.312055 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55e7c0e-8f95-4fbf-8f64-d698998d5f17 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:49.312499 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55e7c0e-8f95-4fbf-8f64-d698998d5f17 None admin] PATH_INFO: `/v3/groups/9ba79133e99b4e91b58e94e32f101d4d` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:49.314398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55e7c0e-8f95-4fbf-8f64-d698998d5f17 None admin] RBAC: Authorizing `identity:get_group(group_id=9ba79133e99b4e91b58e94e32f101d4d)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:49.316942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55e7c0e-8f95-4fbf-8f64-d698998d5f17 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:49.324659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 184/381] 10.222.0.47 () {56 vars in 1180 bytes} [Sun Aug 21 01:04:48 2022] GET /identity/v3/groups/9ba79133e99b4e91b58e94e32f101d4d => generated 226 bytes in 382 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:49.326297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef749b59-50a4-4d81-b0e0-5cca2fb8096a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:49.435788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:49.506271 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef749b59-50a4-4d81-b0e0-5cca2fb8096a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:49.515209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55e7c0e-8f95-4fbf-8f64-d698998d5f17 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:49.540976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:49.572739 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ad1979-d4f9-4351-a79d-d9bcdf2c5139 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:49.574082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad1979-d4f9-4351-a79d-d9bcdf2c5139 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:49.574582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad1979-d4f9-4351-a79d-d9bcdf2c5139 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:49.575056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad1979-d4f9-4351-a79d-d9bcdf2c5139 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:49.576021 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ad1979-d4f9-4351-a79d-d9bcdf2c5139 None admin] RBAC: Adding query filter params (group.id=9ba79133e99b4e91b58e94e32f101d4d, role.id=fbc5961470b74a85bde6be2171f5f08f, scope.project.id=ea5d616ff88c481baff44c2b3840d3d4) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:49.576557 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ad1979-d4f9-4351-a79d-d9bcdf2c5139 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:49.578375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ad1979-d4f9-4351-a79d-d9bcdf2c5139 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:49.593550 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 198/382] 10.222.0.47 () {56 vars in 1376 bytes} [Sun Aug 21 01:04:49 2022] GET /identity/v3/role_assignments?group.id=9ba79133e99b4e91b58e94e32f101d4d&scope.project.id=ea5d616ff88c481baff44c2b3840d3d4&role.id=fbc5961470b74a85bde6be2171f5f08f => generated 260 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:49.691465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55e7c0e-8f95-4fbf-8f64-d698998d5f17 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:49.772335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60409a19-ee37-4f17-80ee-ccb6fc326d4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:49.774398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60409a19-ee37-4f17-80ee-ccb6fc326d4b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:49.775145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60409a19-ee37-4f17-80ee-ccb6fc326d4b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:49.775878 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60409a19-ee37-4f17-80ee-ccb6fc326d4b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:49.791983 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60409a19-ee37-4f17-80ee-ccb6fc326d4b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:49.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-60409a19-ee37-4f17-80ee-ccb6fc326d4b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:49.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:49.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:49.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:49.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:49.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:49.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:49.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:49.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:49.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:49.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:49.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:49.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:49.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:49.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:49.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:49.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:49.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:49.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:49.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:49.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:49.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:49.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:49.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:49.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:49.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:49.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:49.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:49.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:49.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:49.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:49.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:49.799885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:49.799885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:49.799885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:49.799885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:49.799885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:49.799885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:49.799885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:49.799885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:49.799885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:49.799885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:49.799885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:49.799885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:49.799885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 185/383] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:04:49 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:49.809803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ad1979-d4f9-4351-a79d-d9bcdf2c5139 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:49.813029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcaf1a18-be93-4c2a-be65-39a9b2004c57 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:49.813405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcaf1a18-be93-4c2a-be65-39a9b2004c57 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:49.813559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcaf1a18-be93-4c2a-be65-39a9b2004c57 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:49.820149 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 186/384] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:49.823574 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72d6fab-af77-46ee-8c47-996e15559936 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:49.823945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72d6fab-af77-46ee-8c47-996e15559936 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:49.824196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72d6fab-af77-46ee-8c47-996e15559936 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:49.845285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-d72d6fab-af77-46ee-8c47-996e15559936 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:49.846513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d72d6fab-af77-46ee-8c47-996e15559936 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:49.954743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 187/385] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:04:50.007175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ad1979-d4f9-4351-a79d-d9bcdf2c5139 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:50.043880 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:50.057744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cbaa94-0b40-46a3-a5ac-2590f56635c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:50.061635 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cbaa94-0b40-46a3-a5ac-2590f56635c0 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:50.061635 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cbaa94-0b40-46a3-a5ac-2590f56635c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:50.061635 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cbaa94-0b40-46a3-a5ac-2590f56635c0 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:50.061635 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cbaa94-0b40-46a3-a5ac-2590f56635c0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:50.061635 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cbaa94-0b40-46a3-a5ac-2590f56635c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:50.061635 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cbaa94-0b40-46a3-a5ac-2590f56635c0 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:50.076995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 199/386] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:04:49 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:50.083070 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72d6fab-af77-46ee-8c47-996e15559936 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:50.101420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:50.217607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72d6fab-af77-46ee-8c47-996e15559936 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:50.258629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c82a8c-b81d-4eeb-8f83-759c7551df8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:50.259264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c82a8c-b81d-4eeb-8f83-759c7551df8a None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:50.259534 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c82a8c-b81d-4eeb-8f83-759c7551df8a None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:50.259803 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c82a8c-b81d-4eeb-8f83-759c7551df8a None admin] PATH_INFO: `/v3/groups/9ba79133e99b4e91b58e94e32f101d4d` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:50.264885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c82a8c-b81d-4eeb-8f83-759c7551df8a None admin] RBAC: Authorizing `identity:get_group(group_id=9ba79133e99b4e91b58e94e32f101d4d)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:50.266327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c82a8c-b81d-4eeb-8f83-759c7551df8a None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:50.268328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 188/387] 10.222.0.47 () {56 vars in 1180 bytes} [Sun Aug 21 01:04:50 2022] GET /identity/v3/groups/9ba79133e99b4e91b58e94e32f101d4d => generated 226 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:50.274575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cbaa94-0b40-46a3-a5ac-2590f56635c0 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:50.328785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c82a8c-b81d-4eeb-8f83-759c7551df8a None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:50.465785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:50.502529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cbaa94-0b40-46a3-a5ac-2590f56635c0 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:50.524046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c82a8c-b81d-4eeb-8f83-759c7551df8a None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:50.586985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d3e992-40ae-4ec0-8df0-5aa6269dee13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:50.587519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d3e992-40ae-4ec0-8df0-5aa6269dee13 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:50.587622 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d3e992-40ae-4ec0-8df0-5aa6269dee13 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:50.587856 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d3e992-40ae-4ec0-8df0-5aa6269dee13 None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:50.589297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d3e992-40ae-4ec0-8df0-5aa6269dee13 None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:50.591125 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d3e992-40ae-4ec0-8df0-5aa6269dee13 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:50.599303 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 200/388] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:50 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:50.606052 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d3e992-40ae-4ec0-8df0-5aa6269dee13 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:50.611798 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d846e29f-4424-445b-8aca-31edc22eb343 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:50.612490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d846e29f-4424-445b-8aca-31edc22eb343 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:50.612768 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d846e29f-4424-445b-8aca-31edc22eb343 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:50.613051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d846e29f-4424-445b-8aca-31edc22eb343 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:50.626125 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d846e29f-4424-445b-8aca-31edc22eb343 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:50.633250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-d846e29f-4424-445b-8aca-31edc22eb343 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:50.633250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:50.633250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:50.633250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:50.633250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:50.633250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:50.633250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:50.633250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:50.633250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:50.633250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:50.633250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:50.633250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:50.633250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:50.633250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:50.633250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:50.633250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:50.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:50.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:50.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:50.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:50.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:50.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:50.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:50.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:50.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:50.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:50.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:50.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:50.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:50.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:50.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:50.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:50.635028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:50.635028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:50.635028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:50.635028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:50.635028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:50.635028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:50.635028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:50.635028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:50.635028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:50.635028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:50.635028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:50.635028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:50.637011 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 189/389] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:50 2022] GET /identity/v3/roles/member => generated 84 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:04:50.638528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb545475-7d63-4376-8580-f3a0a64174ae None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:50.638813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb545475-7d63-4376-8580-f3a0a64174ae None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:50.639091 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb545475-7d63-4376-8580-f3a0a64174ae None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:50.640863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 190/390] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:50.645672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb545475-7d63-4376-8580-f3a0a64174ae None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:50.717787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:50.771831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d3e992-40ae-4ec0-8df0-5aa6269dee13 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:50.784088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb545475-7d63-4376-8580-f3a0a64174ae None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:50.815683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948a61bb-5c70-4ba9-969a-c119bfe26f7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:50.817126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948a61bb-5c70-4ba9-969a-c119bfe26f7b None admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:50.817808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948a61bb-5c70-4ba9-969a-c119bfe26f7b None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:50.818892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948a61bb-5c70-4ba9-969a-c119bfe26f7b None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4/groups/9ba79133e99b4e91b58e94e32f101d4d/roles/fbc5961470b74a85bde6be2171f5f08f` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:50.821682 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948a61bb-5c70-4ba9-969a-c119bfe26f7b None admin] RBAC: Authorizing `identity:create_grant(project_id=63ed5323666040089b15189fade45bf4, group_id=9ba79133e99b4e91b58e94e32f101d4d, role_id=fbc5961470b74a85bde6be2171f5f08f)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:50.823474 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948a61bb-5c70-4ba9-969a-c119bfe26f7b None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:50.836606 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f10de86-01d6-4d5e-9f9d-1ff98c594587 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:50.837224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f10de86-01d6-4d5e-9f9d-1ff98c594587 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:50.837510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f10de86-01d6-4d5e-9f9d-1ff98c594587 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:50.837765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f10de86-01d6-4d5e-9f9d-1ff98c594587 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:50.838434 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f10de86-01d6-4d5e-9f9d-1ff98c594587 None admin] RBAC: Adding query filter params (name=member) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:50.838738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f10de86-01d6-4d5e-9f9d-1ff98c594587 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:50.841116 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f10de86-01d6-4d5e-9f9d-1ff98c594587 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:50.844219 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:50.859617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 191/391] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:50.860913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51804df8-d840-4b82-b1ec-0fc2ce481ec6 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:50.861211 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51804df8-d840-4b82-b1ec-0fc2ce481ec6 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:50.861468 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51804df8-d840-4b82-b1ec-0fc2ce481ec6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:50.889240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-51804df8-d840-4b82-b1ec-0fc2ce481ec6 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:50.890691 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-51804df8-d840-4b82-b1ec-0fc2ce481ec6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:50.910938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 201/392] 10.222.0.47 () {58 vars in 1442 bytes} [Sun Aug 21 01:04:50 2022] PUT /identity/v3/projects/63ed5323666040089b15189fade45bf4/groups/9ba79133e99b4e91b58e94e32f101d4d/roles/fbc5961470b74a85bde6be2171f5f08f => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:04:50.915447 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948a61bb-5c70-4ba9-969a-c119bfe26f7b None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:50.990776 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 192/393] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:51.145901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948a61bb-5c70-4ba9-969a-c119bfe26f7b None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:51.164996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:51.188727 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9376bd4-b475-4766-8e84-9f3a0d251dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:51.189367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9376bd4-b475-4766-8e84-9f3a0d251dfd None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:51.189634 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9376bd4-b475-4766-8e84-9f3a0d251dfd None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:51.189960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9376bd4-b475-4766-8e84-9f3a0d251dfd None admin] PATH_INFO: `/v3/groups/9ba79133e99b4e91b58e94e32f101d4d` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:51.191188 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9376bd4-b475-4766-8e84-9f3a0d251dfd None admin] RBAC: Authorizing `identity:get_group(group_id=9ba79133e99b4e91b58e94e32f101d4d)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:51.192658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9376bd4-b475-4766-8e84-9f3a0d251dfd None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:51.195476 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 202/394] 10.222.0.47 () {56 vars in 1180 bytes} [Sun Aug 21 01:04:50 2022] GET /identity/v3/groups/9ba79133e99b4e91b58e94e32f101d4d => generated 226 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:51.200969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51804df8-d840-4b82-b1ec-0fc2ce481ec6 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:51.358405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51804df8-d840-4b82-b1ec-0fc2ce481ec6 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:51.364267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9376bd4-b475-4766-8e84-9f3a0d251dfd None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:51.393330 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:51.407952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21e3da6-563f-4981-b7d7-dd9634ebd948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:51.407952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21e3da6-563f-4981-b7d7-dd9634ebd948 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:51.407952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21e3da6-563f-4981-b7d7-dd9634ebd948 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:51.407952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21e3da6-563f-4981-b7d7-dd9634ebd948 None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:51.408400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21e3da6-563f-4981-b7d7-dd9634ebd948 None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:51.409586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21e3da6-563f-4981-b7d7-dd9634ebd948 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:51.411609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 193/395] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:51 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:51.417308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21e3da6-563f-4981-b7d7-dd9634ebd948 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:51.538652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21e3da6-563f-4981-b7d7-dd9634ebd948 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:51.580059 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9376bd4-b475-4766-8e84-9f3a0d251dfd None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:51.582848 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fbd5b3-16af-4039-b808-8e1264aa9356 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:51.583672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fbd5b3-16af-4039-b808-8e1264aa9356 None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:51.583959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fbd5b3-16af-4039-b808-8e1264aa9356 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:51.584260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fbd5b3-16af-4039-b808-8e1264aa9356 None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4/groups/9ba79133e99b4e91b58e94e32f101d4d/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:51.587784 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fbd5b3-16af-4039-b808-8e1264aa9356 None admin] RBAC: Authorizing `identity:create_grant(project_id=63ed5323666040089b15189fade45bf4, group_id=9ba79133e99b4e91b58e94e32f101d4d, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:51.590696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fbd5b3-16af-4039-b808-8e1264aa9356 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:51.636205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e149a4f8-f378-4d90-8bed-9903ecccffa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:51.636860 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e149a4f8-f378-4d90-8bed-9903ecccffa2 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:51.637132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e149a4f8-f378-4d90-8bed-9903ecccffa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:51.637406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e149a4f8-f378-4d90-8bed-9903ecccffa2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:51.650209 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e149a4f8-f378-4d90-8bed-9903ecccffa2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:51.655534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-e149a4f8-f378-4d90-8bed-9903ecccffa2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:51.655534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:51.655534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:51.655534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:51.655534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:51.655534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:51.655534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:51.655534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:51.655534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:51.655534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:51.655534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:51.655534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:51.655534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:51.655534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:51.655534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:51.655534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:51.656601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:51.656601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:51.656601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:51.656601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:51.656601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:51.656601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:51.656601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:51.656601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:51.656601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:51.656601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:51.656601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:51.656601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:51.656601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:51.656601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:51.656601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:51.656601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:51.657425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:51.657425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:51.657425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:51.657425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:51.657425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:51.657425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:51.657425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:51.657425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:51.657425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:51.657425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:51.657425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:51.657425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:51.657425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 203/396] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:51.669357 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 194/397] 10.222.0.47 () {58 vars in 1442 bytes} [Sun Aug 21 01:04:51 2022] PUT /identity/v3/projects/63ed5323666040089b15189fade45bf4/groups/9ba79133e99b4e91b58e94e32f101d4d/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:04:51.669968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e149a4f8-f378-4d90-8bed-9903ecccffa2 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:51.740170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:51.822353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e149a4f8-f378-4d90-8bed-9903ecccffa2 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:51.845389 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfec6c7-3422-4fa6-a460-d8688c2389ba None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:51.845724 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfec6c7-3422-4fa6-a460-d8688c2389ba None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:51.845997 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfec6c7-3422-4fa6-a460-d8688c2389ba None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:51.847524 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 195/398] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:04:51.868001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae1deef-0037-4f95-b817-c90367fc6f6e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:51.868340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae1deef-0037-4f95-b817-c90367fc6f6e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:51.868568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae1deef-0037-4f95-b817-c90367fc6f6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:51.895838 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-8ae1deef-0037-4f95-b817-c90367fc6f6e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:51.897086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8ae1deef-0037-4f95-b817-c90367fc6f6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:51.900663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9312f3a-9216-4e5e-b7ed-6fe4323255c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:51.901295 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9312f3a-9216-4e5e-b7ed-6fe4323255c4 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:51.901809 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9312f3a-9216-4e5e-b7ed-6fe4323255c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:51.902082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9312f3a-9216-4e5e-b7ed-6fe4323255c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:51.902870 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9312f3a-9216-4e5e-b7ed-6fe4323255c4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:51.903178 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9312f3a-9216-4e5e-b7ed-6fe4323255c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:51.904594 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9312f3a-9216-4e5e-b7ed-6fe4323255c4 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:51.918203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 204/399] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:51.924047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9312f3a-9216-4e5e-b7ed-6fe4323255c4 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:51.964765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:51.988378 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 196/400] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:04:52.055929 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9312f3a-9216-4e5e-b7ed-6fe4323255c4 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:52.059601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:52.099788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7645e67-7e10-4976-b484-2a56006d7e62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:52.101027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7645e67-7e10-4976-b484-2a56006d7e62 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:52.101350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7645e67-7e10-4976-b484-2a56006d7e62 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:52.101627 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7645e67-7e10-4976-b484-2a56006d7e62 None admin] PATH_INFO: `/v3/groups/9ba79133e99b4e91b58e94e32f101d4d` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:52.102818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7645e67-7e10-4976-b484-2a56006d7e62 None admin] RBAC: Authorizing `identity:get_group(group_id=9ba79133e99b4e91b58e94e32f101d4d)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:52.104394 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7645e67-7e10-4976-b484-2a56006d7e62 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:52.106354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 205/401] 10.222.0.47 () {56 vars in 1180 bytes} [Sun Aug 21 01:04:51 2022] GET /identity/v3/groups/9ba79133e99b4e91b58e94e32f101d4d => generated 226 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:52.111894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae1deef-0037-4f95-b817-c90367fc6f6e None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:52.240326 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae1deef-0037-4f95-b817-c90367fc6f6e None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:52.285044 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1985b9f2-7183-4180-8f86-ec970a799eb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:52.285696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1985b9f2-7183-4180-8f86-ec970a799eb8 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:52.285973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1985b9f2-7183-4180-8f86-ec970a799eb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:52.286245 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1985b9f2-7183-4180-8f86-ec970a799eb8 None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:52.287388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1985b9f2-7183-4180-8f86-ec970a799eb8 None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:52.289530 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1985b9f2-7183-4180-8f86-ec970a799eb8 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:52.291495 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 197/402] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:52 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:52.304560 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7645e67-7e10-4976-b484-2a56006d7e62 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:52.330693 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00465a7-ffa3-45cd-9ce6-47d2b397294a None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:52.331276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00465a7-ffa3-45cd-9ce6-47d2b397294a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:52.339816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00465a7-ffa3-45cd-9ce6-47d2b397294a None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:52.339816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 198/403] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:52.339816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab72bd8-90ec-4721-8039-1314bbd73aa5 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:52.339816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab72bd8-90ec-4721-8039-1314bbd73aa5 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:52.340498 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab72bd8-90ec-4721-8039-1314bbd73aa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:52.385196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-dab72bd8-90ec-4721-8039-1314bbd73aa5 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:52.386426 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-dab72bd8-90ec-4721-8039-1314bbd73aa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:52.444708 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7645e67-7e10-4976-b484-2a56006d7e62 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:52.476832 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:52.490151 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e7e381-f7f0-4883-80b3-6f65394f5875 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:52.494049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e7e381-f7f0-4883-80b3-6f65394f5875 None admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:52.494049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e7e381-f7f0-4883-80b3-6f65394f5875 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:52.494049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e7e381-f7f0-4883-80b3-6f65394f5875 None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4/groups/9ba79133e99b4e91b58e94e32f101d4d/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:52.494378 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 199/404] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:04:52.494378 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab72bd8-90ec-4721-8039-1314bbd73aa5 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:52.495325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e7e381-f7f0-4883-80b3-6f65394f5875 None admin] RBAC: Authorizing `identity:create_grant(project_id=ea5d616ff88c481baff44c2b3840d3d4, group_id=9ba79133e99b4e91b58e94e32f101d4d, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:52.497006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e7e381-f7f0-4883-80b3-6f65394f5875 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:52.602457 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab72bd8-90ec-4721-8039-1314bbd73aa5 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:52.612057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 206/405] 10.222.0.47 () {58 vars in 1442 bytes} [Sun Aug 21 01:04:52 2022] PUT /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4/groups/9ba79133e99b4e91b58e94e32f101d4d/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:04:52.651986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3732c90-89a0-4009-8cd7-5a1bf01005dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:52.652661 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3732c90-89a0-4009-8cd7-5a1bf01005dd None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:52.652937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3732c90-89a0-4009-8cd7-5a1bf01005dd None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:52.653207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3732c90-89a0-4009-8cd7-5a1bf01005dd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:52.666144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3732c90-89a0-4009-8cd7-5a1bf01005dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:52.671243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-e3732c90-89a0-4009-8cd7-5a1bf01005dd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:52.671243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:52.671243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:52.671243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:52.671243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:52.671243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:52.671243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:52.671243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:52.671243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:52.671243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:52.671243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:52.671243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:52.671243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:52.671243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:52.671243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:52.671243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:52.672247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:52.672247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:52.672247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:52.672247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:52.672247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:52.672247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:52.672247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:52.672247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:52.672247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:52.672247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:52.672247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:52.672247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:52.672247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:52.672247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:52.672247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:52.672247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:52.673372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:52.673372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:52.673372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:52.673372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:52.673372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:52.673372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:52.673372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:52.673372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:52.673372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:52.673372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:52.673372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:52.673372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:52.673372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 200/406] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:04:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:52.680375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e7e381-f7f0-4883-80b3-6f65394f5875 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:52.746094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:52.841849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e7e381-f7f0-4883-80b3-6f65394f5875 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:52.881369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee50a45-8ff3-49d5-8349-2c8e3e150202 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:52.882284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee50a45-8ff3-49d5-8349-2c8e3e150202 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:52.882708 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee50a45-8ff3-49d5-8349-2c8e3e150202 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:52.883146 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee50a45-8ff3-49d5-8349-2c8e3e150202 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:52.883977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee50a45-8ff3-49d5-8349-2c8e3e150202 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:52.884440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee50a45-8ff3-49d5-8349-2c8e3e150202 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:52.886055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee50a45-8ff3-49d5-8349-2c8e3e150202 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:52.896019 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 207/407] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:52.902254 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3732c90-89a0-4009-8cd7-5a1bf01005dd None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:52.909853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee50a45-8ff3-49d5-8349-2c8e3e150202 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:53.004119 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:53.042893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:53.054016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee50a45-8ff3-49d5-8349-2c8e3e150202 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:53.108872 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2428b518-8d48-4a21-8673-d7ded1cc0946 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:53.110620 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428b518-8d48-4a21-8673-d7ded1cc0946 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:53.112378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428b518-8d48-4a21-8673-d7ded1cc0946 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:53.112922 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428b518-8d48-4a21-8673-d7ded1cc0946 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:53.139786 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2428b518-8d48-4a21-8673-d7ded1cc0946 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:53.148133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3732c90-89a0-4009-8cd7-5a1bf01005dd None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:53.156834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-2428b518-8d48-4a21-8673-d7ded1cc0946 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:53.156834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:53.156834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:53.156834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:53.156834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:53.156834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:53.156834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:53.156834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:53.156834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:53.156834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:53.156834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:53.156834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:53.156834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:53.156834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:53.156834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:53.156834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:53.157789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:53.157789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:53.157789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:53.157789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:53.157789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:53.157789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:53.157789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:53.157789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:53.157789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:53.157789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:53.157789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:53.157789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:53.157789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:53.157789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:53.157789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:53.157789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:53.158604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:53.158604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:53.158604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:53.158604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:53.158604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:53.158604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:53.158604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:53.158604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:53.158604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:53.158604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:53.158604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:53.158604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:53.158604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 208/408] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:04:52 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:04:53.169148 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2428b518-8d48-4a21-8673-d7ded1cc0946 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:53.197145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f85b632-0128-4df5-a6ce-c00bbe928e0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:53.198146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f85b632-0128-4df5-a6ce-c00bbe928e0d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:53.198616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f85b632-0128-4df5-a6ce-c00bbe928e0d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:53.199165 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f85b632-0128-4df5-a6ce-c00bbe928e0d None admin] PATH_INFO: `/v3/groups/22c4cfab6749435b9afdacfff854dd85` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:53.200494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f85b632-0128-4df5-a6ce-c00bbe928e0d None admin] RBAC: Authorizing `identity:get_group(group_id=22c4cfab6749435b9afdacfff854dd85)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:53.202302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f85b632-0128-4df5-a6ce-c00bbe928e0d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:53.204710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 201/409] 10.222.0.47 () {56 vars in 1180 bytes} [Sun Aug 21 01:04:52 2022] GET /identity/v3/groups/22c4cfab6749435b9afdacfff854dd85 => generated 229 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:53.214110 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f85b632-0128-4df5-a6ce-c00bbe928e0d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:53.324720 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2428b518-8d48-4a21-8673-d7ded1cc0946 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:53.369852 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f85b632-0128-4df5-a6ce-c00bbe928e0d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:53.375261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8c352e-dc59-4b32-b762-c4426cbb172f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:53.375889 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8c352e-dc59-4b32-b762-c4426cbb172f None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:53.376187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8c352e-dc59-4b32-b762-c4426cbb172f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:53.376459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8c352e-dc59-4b32-b762-c4426cbb172f None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:53.377687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8c352e-dc59-4b32-b762-c4426cbb172f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:53.388149 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8c352e-dc59-4b32-b762-c4426cbb172f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:53.388149 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8c352e-dc59-4b32-b762-c4426cbb172f None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:53.395204 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:53.401747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 209/410] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:04:53 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:53.405364 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8c352e-dc59-4b32-b762-c4426cbb172f None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:53.444711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:53.450109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4240f35-c097-4d2b-b948-3857ddf758c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:53.450753 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4240f35-c097-4d2b-b948-3857ddf758c7 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:53.451026 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4240f35-c097-4d2b-b948-3857ddf758c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:53.451295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4240f35-c097-4d2b-b948-3857ddf758c7 None admin] PATH_INFO: `/v3/projects/ada79d28652f4a3aa0a174133f24108f` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:53.454677 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4240f35-c097-4d2b-b948-3857ddf758c7 None admin] RBAC: Authorizing `identity:get_project(project_id=ada79d28652f4a3aa0a174133f24108f)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:53.462579 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4240f35-c097-4d2b-b948-3857ddf758c7 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:53.462579 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 202/411] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:53 2022] GET /identity/v3/projects/ada79d28652f4a3aa0a174133f24108f => generated 344 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:53.467065 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4240f35-c097-4d2b-b948-3857ddf758c7 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:53.573574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8c352e-dc59-4b32-b762-c4426cbb172f None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:53.591307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4240f35-c097-4d2b-b948-3857ddf758c7 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:53.611468 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b421ff-78ef-4a8e-9e68-b80eeac9e0c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:53.612562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b421ff-78ef-4a8e-9e68-b80eeac9e0c5 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:53.613036 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b421ff-78ef-4a8e-9e68-b80eeac9e0c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:53.613499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b421ff-78ef-4a8e-9e68-b80eeac9e0c5 None admin] PATH_INFO: `/v3/groups/9ba79133e99b4e91b58e94e32f101d4d` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:53.614890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b421ff-78ef-4a8e-9e68-b80eeac9e0c5 None admin] RBAC: Authorizing `identity:get_group(group_id=9ba79133e99b4e91b58e94e32f101d4d)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:53.617202 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b421ff-78ef-4a8e-9e68-b80eeac9e0c5 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:53.619406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 210/412] 10.222.0.47 () {56 vars in 1180 bytes} [Sun Aug 21 01:04:53 2022] GET /identity/v3/groups/9ba79133e99b4e91b58e94e32f101d4d => generated 226 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:53.620843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdbe337-85b6-4887-a81f-1572e333d529 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:53.621324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdbe337-85b6-4887-a81f-1572e333d529 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:53.621834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdbe337-85b6-4887-a81f-1572e333d529 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:53.627543 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 211/413] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:53 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:53.628698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdbe337-85b6-4887-a81f-1572e333d529 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:53.634898 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b04f81a-0b6f-4bf2-bc71-4c37c42e341e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:53.635557 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b04f81a-0b6f-4bf2-bc71-4c37c42e341e None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:53.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b04f81a-0b6f-4bf2-bc71-4c37c42e341e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:53.636137 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b04f81a-0b6f-4bf2-bc71-4c37c42e341e None admin] PATH_INFO: `/v3/projects/ada79d28652f4a3aa0a174133f24108f/groups/22c4cfab6749435b9afdacfff854dd85/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:53.638039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b04f81a-0b6f-4bf2-bc71-4c37c42e341e None admin] RBAC: Authorizing `identity:create_grant(project_id=ada79d28652f4a3aa0a174133f24108f, group_id=22c4cfab6749435b9afdacfff854dd85, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:53.639463 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b04f81a-0b6f-4bf2-bc71-4c37c42e341e None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:53.727605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 203/414] 10.222.0.47 () {58 vars in 1442 bytes} [Sun Aug 21 01:04:53 2022] PUT /identity/v3/projects/ada79d28652f4a3aa0a174133f24108f/groups/22c4cfab6749435b9afdacfff854dd85/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:04:53.729762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ce5881-88af-4371-8409-e7a335bbb737 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:53.730039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ce5881-88af-4371-8409-e7a335bbb737 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:53.730371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ce5881-88af-4371-8409-e7a335bbb737 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:53.739789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:53.754080 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdbe337-85b6-4887-a81f-1572e333d529 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:53.773302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-e9ce5881-88af-4371-8409-e7a335bbb737 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:53.777263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e9ce5881-88af-4371-8409-e7a335bbb737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:53.807071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a4e26c-dcd9-4709-ae0e-7ce832c1ea90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:53.807727 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a4e26c-dcd9-4709-ae0e-7ce832c1ea90 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:53.808000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a4e26c-dcd9-4709-ae0e-7ce832c1ea90 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:53.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a4e26c-dcd9-4709-ae0e-7ce832c1ea90 None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:53.810711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a4e26c-dcd9-4709-ae0e-7ce832c1ea90 None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:53.812258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a4e26c-dcd9-4709-ae0e-7ce832c1ea90 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:53.816982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 212/415] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:53 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:53.827846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a4e26c-dcd9-4709-ae0e-7ce832c1ea90 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:53.835302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:53.902705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 204/416] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:54.013554 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97672a26-1e61-426a-ba56-26cdc802b160 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:54.013898 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97672a26-1e61-426a-ba56-26cdc802b160 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:54.014106 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97672a26-1e61-426a-ba56-26cdc802b160 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:54.015614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 205/417] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:04:54.021103 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a4e26c-dcd9-4709-ae0e-7ce832c1ea90 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:54.021279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4ba408-df9c-475d-bcd2-12ea53d94e8b None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:54.021461 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4ba408-df9c-475d-bcd2-12ea53d94e8b None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:54.021707 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4ba408-df9c-475d-bcd2-12ea53d94e8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:54.048376 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-5f4ba408-df9c-475d-bcd2-12ea53d94e8b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:54.049585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5f4ba408-df9c-475d-bcd2-12ea53d94e8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:54.058684 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae65ce0-6b35-488b-bc94-9092c7b45383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:54.059355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae65ce0-6b35-488b-bc94-9092c7b45383 None admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:54.059631 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae65ce0-6b35-488b-bc94-9092c7b45383 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:54.059900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae65ce0-6b35-488b-bc94-9092c7b45383 None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4/groups/9ba79133e99b4e91b58e94e32f101d4d/roles/fbc5961470b74a85bde6be2171f5f08f` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:54.062169 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae65ce0-6b35-488b-bc94-9092c7b45383 None admin] RBAC: Authorizing `identity:create_grant(project_id=ea5d616ff88c481baff44c2b3840d3d4, group_id=9ba79133e99b4e91b58e94e32f101d4d, role_id=fbc5961470b74a85bde6be2171f5f08f)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:54.063630 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae65ce0-6b35-488b-bc94-9092c7b45383 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:54.141387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 213/418] 10.222.0.47 () {58 vars in 1442 bytes} [Sun Aug 21 01:04:53 2022] PUT /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4/groups/9ba79133e99b4e91b58e94e32f101d4d/roles/fbc5961470b74a85bde6be2171f5f08f => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:04:54.144782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 206/419] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:04:54.219821 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae65ce0-6b35-488b-bc94-9092c7b45383 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:54.238190 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:54.337242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae65ce0-6b35-488b-bc94-9092c7b45383 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:54.377848 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eecd73e-b76b-4680-8520-92ed7353ab2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:54.378543 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eecd73e-b76b-4680-8520-92ed7353ab2f None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:54.378818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eecd73e-b76b-4680-8520-92ed7353ab2f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:54.379085 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eecd73e-b76b-4680-8520-92ed7353ab2f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:54.391197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eecd73e-b76b-4680-8520-92ed7353ab2f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:54.396643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-8eecd73e-b76b-4680-8520-92ed7353ab2f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:54.396643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:54.396643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:54.396643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:54.396643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:54.396643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:54.396643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:54.396643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:54.396643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:54.396643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:54.396643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:54.396643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:54.396643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:54.396643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:54.396643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:54.396643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:54.398152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:54.398152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:54.398152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:54.398152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:54.398152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:54.398152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:54.398152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:54.398152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:54.398152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:54.398152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:54.398152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:54.398152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:54.398152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:54.398152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:54.398152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:54.398152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:54.398959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:54.398959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:54.398959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:54.398959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:54.398959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:54.398959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:54.398959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:54.398959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:54.398959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:54.398959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:54.398959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:54.398959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:54.398959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 214/420] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:04:54 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:54.407965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4ba408-df9c-475d-bcd2-12ea53d94e8b None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:54.531058 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:54.535886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4ba408-df9c-475d-bcd2-12ea53d94e8b None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:54.548544 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eecd73e-b76b-4680-8520-92ed7353ab2f None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:54.595023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8bf333-0268-479f-aff2-2316fb6b91ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:54.596450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8bf333-0268-479f-aff2-2316fb6b91ec None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:54.597885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8bf333-0268-479f-aff2-2316fb6b91ec None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:54.598363 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8bf333-0268-479f-aff2-2316fb6b91ec None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:54.599973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8bf333-0268-479f-aff2-2316fb6b91ec None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:54.601263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8bf333-0268-479f-aff2-2316fb6b91ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:54.604942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8bf333-0268-479f-aff2-2316fb6b91ec None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:54.627643 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 207/421] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:04:54 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:54.629093 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8bf333-0268-479f-aff2-2316fb6b91ec None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:54.692904 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:54.729029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eecd73e-b76b-4680-8520-92ed7353ab2f None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:54.770774 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90b615c-c4fa-40ea-bea9-521ea5aef45f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:54.771434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90b615c-c4fa-40ea-bea9-521ea5aef45f None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:54.771677 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90b615c-c4fa-40ea-bea9-521ea5aef45f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:54.772051 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90b615c-c4fa-40ea-bea9-521ea5aef45f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:54.784588 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90b615c-c4fa-40ea-bea9-521ea5aef45f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:54.791299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-b90b615c-c4fa-40ea-bea9-521ea5aef45f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:54.791299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:54.791299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:54.791299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:54.791299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:54.791299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:54.791299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:54.791299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:54.791299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:54.791299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:54.791299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:54.791299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:54.791299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:54.791299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:54.791299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:54.791299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:54.792290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:54.792290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:54.792290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:54.792290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:54.792290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:54.792290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:54.792290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:54.792290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:54.792290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:54.792290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:54.792290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:54.792290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:54.792290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:54.792290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:54.792290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:54.792290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:54.793094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:54.793094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:54.793094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:54.793094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:54.793094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:54.793094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:54.793094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:54.793094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:54.793094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:54.793094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:54.793094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:54.793094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:54.793094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 215/422] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:04:54.800508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90b615c-c4fa-40ea-bea9-521ea5aef45f None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:54.928839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90b615c-c4fa-40ea-bea9-521ea5aef45f None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:54.937487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8bf333-0268-479f-aff2-2316fb6b91ec None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:54.976479 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:54.986014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f83cf34-4d28-49b0-b4aa-1f8221b1dc39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:54.986639 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f83cf34-4d28-49b0-b4aa-1f8221b1dc39 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:54.986917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f83cf34-4d28-49b0-b4aa-1f8221b1dc39 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:54.992931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f83cf34-4d28-49b0-b4aa-1f8221b1dc39 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:54.992931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f83cf34-4d28-49b0-b4aa-1f8221b1dc39 None admin] RBAC: Adding query filter params (name=member) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:54.992931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f83cf34-4d28-49b0-b4aa-1f8221b1dc39 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:54.992931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f83cf34-4d28-49b0-b4aa-1f8221b1dc39 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:55.010652 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 216/423] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:55.014187 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbc8a8a-72ed-4272-b06a-fc2a29b52619 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:55.014927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbc8a8a-72ed-4272-b06a-fc2a29b52619 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:55.016128 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f83cf34-4d28-49b0-b4aa-1f8221b1dc39 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:55.016823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbc8a8a-72ed-4272-b06a-fc2a29b52619 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:55.017100 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbc8a8a-72ed-4272-b06a-fc2a29b52619 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:55.054751 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:55.107712 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbc8a8a-72ed-4272-b06a-fc2a29b52619 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:55.109301 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbc8a8a-72ed-4272-b06a-fc2a29b52619 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:55.170267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f83cf34-4d28-49b0-b4aa-1f8221b1dc39 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:55.224799 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 208/424] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 596 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 21 01:04:55.226793 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d92ca2c-5ba6-4aea-a8b6-abab65969770 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:55.226912 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8450ec4-72fd-4c51-9017-1e699f0210d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:55.227078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d92ca2c-5ba6-4aea-a8b6-abab65969770 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:55.227417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d92ca2c-5ba6-4aea-a8b6-abab65969770 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:55.227988 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8450ec4-72fd-4c51-9017-1e699f0210d0 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:55.228437 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8450ec4-72fd-4c51-9017-1e699f0210d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:55.228890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8450ec4-72fd-4c51-9017-1e699f0210d0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:55.233043 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 209/425] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:55.234086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d92ca2c-5ba6-4aea-a8b6-abab65969770 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:55.306673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:55.328196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:55.349721 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8450ec4-72fd-4c51-9017-1e699f0210d0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:55.351654 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8450ec4-72fd-4c51-9017-1e699f0210d0 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:55.366728 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d92ca2c-5ba6-4aea-a8b6-abab65969770 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:55.413469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0648fdbd-9cf6-4042-835c-34fa8a60abf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:55.414970 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0648fdbd-9cf6-4042-835c-34fa8a60abf3 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:55.415694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0648fdbd-9cf6-4042-835c-34fa8a60abf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:55.416194 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0648fdbd-9cf6-4042-835c-34fa8a60abf3 None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:55.418240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0648fdbd-9cf6-4042-835c-34fa8a60abf3 None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:55.419879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0648fdbd-9cf6-4042-835c-34fa8a60abf3 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:55.422557 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 210/426] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:55 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:55.425364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfc533f-c42d-4419-9f15-24eafda1c91f None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:55.426432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfc533f-c42d-4419-9f15-24eafda1c91f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:55.427133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfc533f-c42d-4419-9f15-24eafda1c91f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:55.454592 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-3dfc533f-c42d-4419-9f15-24eafda1c91f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:55.457723 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-3dfc533f-c42d-4419-9f15-24eafda1c91f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:55.493654 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 217/427] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 478 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:55.495679 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8450ec4-72fd-4c51-9017-1e699f0210d0 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:55.549089 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 211/428] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:55.556754 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfc533f-c42d-4419-9f15-24eafda1c91f None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:55.670113 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:55.672330 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:55.686820 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8450ec4-72fd-4c51-9017-1e699f0210d0 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:55.744744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfc533f-c42d-4419-9f15-24eafda1c91f None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:55.759551 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e45110f-b33f-4aeb-8cbc-0d59f43cd145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:55.760225 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e45110f-b33f-4aeb-8cbc-0d59f43cd145 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:55.760503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e45110f-b33f-4aeb-8cbc-0d59f43cd145 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:55.760773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e45110f-b33f-4aeb-8cbc-0d59f43cd145 None admin] PATH_INFO: `/v3/groups/9ba79133e99b4e91b58e94e32f101d4d` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:55.763960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e45110f-b33f-4aeb-8cbc-0d59f43cd145 None admin] RBAC: Authorizing `identity:get_group(group_id=9ba79133e99b4e91b58e94e32f101d4d)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:55.765519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e45110f-b33f-4aeb-8cbc-0d59f43cd145 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:55.768881 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 218/429] 10.222.0.47 () {56 vars in 1180 bytes} [Sun Aug 21 01:04:55 2022] GET /identity/v3/groups/9ba79133e99b4e91b58e94e32f101d4d => generated 226 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:55.774081 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e45110f-b33f-4aeb-8cbc-0d59f43cd145 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:55.819349 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e12e5b-a026-4622-8496-cc81d111ee61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:55.819927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e12e5b-a026-4622-8496-cc81d111ee61 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:55.820232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e12e5b-a026-4622-8496-cc81d111ee61 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:55.820502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e12e5b-a026-4622-8496-cc81d111ee61 None admin] PATH_INFO: `/v3/projects/63ed5323666040089b15189fade45bf4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:55.822167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e12e5b-a026-4622-8496-cc81d111ee61 None admin] RBAC: Authorizing `identity:get_project(project_id=63ed5323666040089b15189fade45bf4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:55.823673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e12e5b-a026-4622-8496-cc81d111ee61 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:55.826349 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 212/430] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:55 2022] GET /identity/v3/projects/63ed5323666040089b15189fade45bf4 => generated 298 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:55.831529 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e12e5b-a026-4622-8496-cc81d111ee61 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:55.963235 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e45110f-b33f-4aeb-8cbc-0d59f43cd145 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:55.990217 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e12e5b-a026-4622-8496-cc81d111ee61 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:56.003189 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c86aab-4280-4572-9158-9e5a91b9d3e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:56.003805 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c86aab-4280-4572-9158-9e5a91b9d3e1 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:56.004083 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c86aab-4280-4572-9158-9e5a91b9d3e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:56.004381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c86aab-4280-4572-9158-9e5a91b9d3e1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:56.005116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c86aab-4280-4572-9158-9e5a91b9d3e1 None admin] RBAC: Adding query filter params (group.id=9ba79133e99b4e91b58e94e32f101d4d, role.id=fbc5961470b74a85bde6be2171f5f08f, scope.project.id=63ed5323666040089b15189fade45bf4) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:56.005422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c86aab-4280-4572-9158-9e5a91b9d3e1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:56.008252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c86aab-4280-4572-9158-9e5a91b9d3e1 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:56.015897 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 219/431] 10.222.0.47 () {56 vars in 1376 bytes} [Sun Aug 21 01:04:55 2022] GET /identity/v3/role_assignments?group.id=9ba79133e99b4e91b58e94e32f101d4d&scope.project.id=63ed5323666040089b15189fade45bf4&role.id=fbc5961470b74a85bde6be2171f5f08f => generated 611 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:56.016894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c86aab-4280-4572-9158-9e5a91b9d3e1 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:56.025138 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:56.036418 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694d851f-89be-4212-9731-6b436325ec3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:56.042535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694d851f-89be-4212-9731-6b436325ec3a None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:56.042535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694d851f-89be-4212-9731-6b436325ec3a None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:56.042535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694d851f-89be-4212-9731-6b436325ec3a None admin] PATH_INFO: `/v3/groups/9ba79133e99b4e91b58e94e32f101d4d` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:56.042535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694d851f-89be-4212-9731-6b436325ec3a None admin] RBAC: Authorizing `identity:get_group(group_id=9ba79133e99b4e91b58e94e32f101d4d)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:56.042535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694d851f-89be-4212-9731-6b436325ec3a None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:56.045403 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 213/432] 10.222.0.47 () {56 vars in 1180 bytes} [Sun Aug 21 01:04:55 2022] GET /identity/v3/groups/9ba79133e99b4e91b58e94e32f101d4d => generated 226 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:56.050947 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694d851f-89be-4212-9731-6b436325ec3a None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:56.057121 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:56.161842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c86aab-4280-4572-9158-9e5a91b9d3e1 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:56.196279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694d851f-89be-4212-9731-6b436325ec3a None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:56.214146 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053d25f7-fa83-4fdf-bb20-8cb410ad1bf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:56.215452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053d25f7-fa83-4fdf-bb20-8cb410ad1bf3 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:56.215949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053d25f7-fa83-4fdf-bb20-8cb410ad1bf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:56.216598 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053d25f7-fa83-4fdf-bb20-8cb410ad1bf3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:56.232170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053d25f7-fa83-4fdf-bb20-8cb410ad1bf3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:56.239078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc4d247-1bff-4241-9b93-823edf92e1e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:56.239691 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc4d247-1bff-4241-9b93-823edf92e1e0 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:56.239989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc4d247-1bff-4241-9b93-823edf92e1e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:56.240338 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc4d247-1bff-4241-9b93-823edf92e1e0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:56.241633 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc4d247-1bff-4241-9b93-823edf92e1e0 None admin] RBAC: Adding query filter params (group.id=9ba79133e99b4e91b58e94e32f101d4d, role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe, scope.project.id=63ed5323666040089b15189fade45bf4) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:56.241938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc4d247-1bff-4241-9b93-823edf92e1e0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:56.242030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-053d25f7-fa83-4fdf-bb20-8cb410ad1bf3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:56.242030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:56.242030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:56.242030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:56.242030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:56.242030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:56.242030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:56.242030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:56.242030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:56.242030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:56.242030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:56.242030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:56.242030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:56.242030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:56.242030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:56.242030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:56.242851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:56.242851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:56.242851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:56.242851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:56.242851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:56.242851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:56.242851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:56.242851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:56.242851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:56.242851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:56.242851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:56.242851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:56.242851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:56.242851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:56.242851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:56.242851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:56.243636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:56.243636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:56.243636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:56.243636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:56.243636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:56.243636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:56.243636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:56.243636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:56.243636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:56.243636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:56.243636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 21 01:04:56.243636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:56.244295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc4d247-1bff-4241-9b93-823edf92e1e0 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:56.244362 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 220/433] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:04:56.245041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d26e13a-2169-404f-b93c-9927d446f1fb None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:56.245041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d26e13a-2169-404f-b93c-9927d446f1fb None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:56.245276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d26e13a-2169-404f-b93c-9927d446f1fb None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:56.246674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 221/434] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:56.251621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d26e13a-2169-404f-b93c-9927d446f1fb None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:56.256409 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 214/435] 10.222.0.47 () {56 vars in 1376 bytes} [Sun Aug 21 01:04:56 2022] GET /identity/v3/role_assignments?group.id=9ba79133e99b4e91b58e94e32f101d4d&scope.project.id=63ed5323666040089b15189fade45bf4&role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 611 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:56.257327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc1d8e8-2259-4dc2-8d54-a91472319927 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:56.257616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc1d8e8-2259-4dc2-8d54-a91472319927 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:56.257888 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc1d8e8-2259-4dc2-8d54-a91472319927 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:56.278171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-5dc1d8e8-2259-4dc2-8d54-a91472319927 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:56.279488 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5dc1d8e8-2259-4dc2-8d54-a91472319927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:56.355592 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:56.379085 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 215/436] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:04:56.393338 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:56.411957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d26e13a-2169-404f-b93c-9927d446f1fb None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:56.454342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29bbd65-1195-4e15-b182-230ab6e03438 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:56.455332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29bbd65-1195-4e15-b182-230ab6e03438 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:56.455803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29bbd65-1195-4e15-b182-230ab6e03438 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:56.456303 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29bbd65-1195-4e15-b182-230ab6e03438 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:56.457357 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29bbd65-1195-4e15-b182-230ab6e03438 None admin] RBAC: Adding query filter params (name=member) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:56.458158 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29bbd65-1195-4e15-b182-230ab6e03438 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:56.460553 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29bbd65-1195-4e15-b182-230ab6e03438 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:56.471000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 222/437] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:04:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:56.476967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc1d8e8-2259-4dc2-8d54-a91472319927 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:56.485644 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d18656-d2c3-46f8-8106-3c1418d8aa8f None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:56.485986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d18656-d2c3-46f8-8106-3c1418d8aa8f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:56.486256 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d18656-d2c3-46f8-8106-3c1418d8aa8f None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:56.487799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 223/438] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:04:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:04:56.493970 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e223f1-a4dd-4ecc-a425-a57150afbe5b None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:56.494247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e223f1-a4dd-4ecc-a425-a57150afbe5b None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:56.494515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e223f1-a4dd-4ecc-a425-a57150afbe5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:56.518150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-67e223f1-a4dd-4ecc-a425-a57150afbe5b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:04:56.519392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-67e223f1-a4dd-4ecc-a425-a57150afbe5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:04:56.609430 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 224/439] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:04:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:04:56.613508 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc1d8e8-2259-4dc2-8d54-a91472319927 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:56.653123 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3283c5d5-fc54-4300-b5e3-e65cd91d7d98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:56.653798 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3283c5d5-fc54-4300-b5e3-e65cd91d7d98 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:56.654077 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3283c5d5-fc54-4300-b5e3-e65cd91d7d98 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:56.654347 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3283c5d5-fc54-4300-b5e3-e65cd91d7d98 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:56.660886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e223f1-a4dd-4ecc-a425-a57150afbe5b None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:56.732698 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:56.748901 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3283c5d5-fc54-4300-b5e3-e65cd91d7d98 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:56.750505 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3283c5d5-fc54-4300-b5e3-e65cd91d7d98 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:56.763202 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:56.792542 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e223f1-a4dd-4ecc-a425-a57150afbe5b None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:56.839012 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bf6d85-a104-4eb1-8045-636a5b8a5ef0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:56.840049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bf6d85-a104-4eb1-8045-636a5b8a5ef0 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:56.840598 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bf6d85-a104-4eb1-8045-636a5b8a5ef0 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:56.841740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bf6d85-a104-4eb1-8045-636a5b8a5ef0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:56.860041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8bf6d85-a104-4eb1-8045-636a5b8a5ef0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:56.864915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-c8bf6d85-a104-4eb1-8045-636a5b8a5ef0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:56.864915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:56.864915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:56.864915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:56.864915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:56.864915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:56.864915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:56.864915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:56.864915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:56.864915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:56.864915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:56.864915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:56.864915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:56.864915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:56.864915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:56.864915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:56.866382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:56.866382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:56.866382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:56.866382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:56.866382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:56.866382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:56.866382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:56.866382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:56.866382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:56.866382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:56.866382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:56.866382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:56.866382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:56.866382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:56.866382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:56.866382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:56.867198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:56.867198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:56.867198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:56.867198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:56.867198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:56.867198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:56.867198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:56.867198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:56.867198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:56.867198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:56.867198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:04:56.867198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:04:56.867198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 225/440] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:04:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:04:56.873921 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bf6d85-a104-4eb1-8045-636a5b8a5ef0 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:56.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 216/441] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:56.897631 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3283c5d5-fc54-4300-b5e3-e65cd91d7d98 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:57.011108 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bf6d85-a104-4eb1-8045-636a5b8a5ef0 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:57.012138 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:57.027933 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3283c5d5-fc54-4300-b5e3-e65cd91d7d98 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:57.050790 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ab22cf-d673-43a2-ba17-bd34b069ff45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:57.051394 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ab22cf-d673-43a2-ba17-bd34b069ff45 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:57.051672 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ab22cf-d673-43a2-ba17-bd34b069ff45 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:57.051939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ab22cf-d673-43a2-ba17-bd34b069ff45 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:57.052630 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ab22cf-d673-43a2-ba17-bd34b069ff45 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:57.052957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ab22cf-d673-43a2-ba17-bd34b069ff45 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:57.054336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ab22cf-d673-43a2-ba17-bd34b069ff45 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:57.059665 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:57.063077 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 226/442] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:04:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:57.064552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effbef08-c487-4fad-a802-f1c58e67f2e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:57.065022 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ab22cf-d673-43a2-ba17-bd34b069ff45 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:57.065187 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effbef08-c487-4fad-a802-f1c58e67f2e3 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:57.065460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effbef08-c487-4fad-a802-f1c58e67f2e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:57.074214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effbef08-c487-4fad-a802-f1c58e67f2e3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:57.078141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effbef08-c487-4fad-a802-f1c58e67f2e3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:04:57.082921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-effbef08-c487-4fad-a802-f1c58e67f2e3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:57.082921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:04:57.082921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:04:57.082921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:04:57.082921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:04:57.082921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:04:57.082921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:04:57.082921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:04:57.082921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:04:57.082921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:04:57.082921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:04:57.082921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:04:57.082921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:04:57.082921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:04:57.082921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:04:57.082921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:04:57.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:04:57.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:04:57.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:04:57.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:04:57.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:04:57.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:04:57.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:04:57.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:04:57.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:04:57.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:04:57.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:04:57.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:04:57.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:04:57.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:04:57.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:04:57.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:04:57.085620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:04:57.085620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:04:57.085620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:04:57.085620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:04:57.085620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:04:57.085620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:04:57.085620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:04:57.085620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:04:57.085620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:04:57.085620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:04:57.085620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:04:57.085620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:04:57.085620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 217/443] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:04:56 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:04:57.085620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effbef08-c487-4fad-a802-f1c58e67f2e3 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:57.170926 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ab22cf-d673-43a2-ba17-bd34b069ff45 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:57.189858 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effbef08-c487-4fad-a802-f1c58e67f2e3 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:57.207758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c2819a-cf5d-434f-a6c1-5d6f861cdb70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:57.208433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c2819a-cf5d-434f-a6c1-5d6f861cdb70 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:57.208705 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c2819a-cf5d-434f-a6c1-5d6f861cdb70 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:57.208976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c2819a-cf5d-434f-a6c1-5d6f861cdb70 None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:57.210765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c2819a-cf5d-434f-a6c1-5d6f861cdb70 None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:57.212239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c2819a-cf5d-434f-a6c1-5d6f861cdb70 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:57.213894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 227/444] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:57 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:57.214676 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c2819a-cf5d-434f-a6c1-5d6f861cdb70 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:57.225867 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bc26c7-568d-42b8-b375-b84b7c00f5ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:57.226477 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bc26c7-568d-42b8-b375-b84b7c00f5ad None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:57.226755 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bc26c7-568d-42b8-b375-b84b7c00f5ad None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:57.227026 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bc26c7-568d-42b8-b375-b84b7c00f5ad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:57.245287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:57.283139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:57.298061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bc26c7-568d-42b8-b375-b84b7c00f5ad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:57.299508 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bc26c7-568d-42b8-b375-b84b7c00f5ad None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:57.317444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c2819a-cf5d-434f-a6c1-5d6f861cdb70 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:57.353845 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9055086-bec1-4715-9965-46a77a9ca863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:57.354451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9055086-bec1-4715-9965-46a77a9ca863 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:57.354725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9055086-bec1-4715-9965-46a77a9ca863 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:57.355015 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9055086-bec1-4715-9965-46a77a9ca863 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:57.355673 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9055086-bec1-4715-9965-46a77a9ca863 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:57.355969 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9055086-bec1-4715-9965-46a77a9ca863 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:57.357384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9055086-bec1-4715-9965-46a77a9ca863 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:57.366890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 228/445] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:04:57 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:57.368960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9055086-bec1-4715-9965-46a77a9ca863 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:57.413931 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 218/446] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:57.414790 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bc26c7-568d-42b8-b375-b84b7c00f5ad None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:57.466772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:57.479052 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9055086-bec1-4715-9965-46a77a9ca863 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:57.510832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:57.517461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac09fff0-f6bd-4800-91f3-0a74b9e17b82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:57.518103 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac09fff0-f6bd-4800-91f3-0a74b9e17b82 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:57.518377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac09fff0-f6bd-4800-91f3-0a74b9e17b82 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:57.518649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac09fff0-f6bd-4800-91f3-0a74b9e17b82 None admin] PATH_INFO: `/v3/groups/9ba79133e99b4e91b58e94e32f101d4d` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:57.519812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac09fff0-f6bd-4800-91f3-0a74b9e17b82 None admin] RBAC: Authorizing `identity:get_group(group_id=9ba79133e99b4e91b58e94e32f101d4d)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:57.521282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac09fff0-f6bd-4800-91f3-0a74b9e17b82 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:57.522872 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 229/447] 10.222.0.47 () {56 vars in 1180 bytes} [Sun Aug 21 01:04:57 2022] GET /identity/v3/groups/9ba79133e99b4e91b58e94e32f101d4d => generated 226 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:57.523640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac09fff0-f6bd-4800-91f3-0a74b9e17b82 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:57.523799 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bc26c7-568d-42b8-b375-b84b7c00f5ad None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:57.560304 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1770b1aa-07b5-46cf-b6b1-79ab078b38dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:57.560930 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1770b1aa-07b5-46cf-b6b1-79ab078b38dc None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:57.561213 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1770b1aa-07b5-46cf-b6b1-79ab078b38dc None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:57.561476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1770b1aa-07b5-46cf-b6b1-79ab078b38dc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:57.629522 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac09fff0-f6bd-4800-91f3-0a74b9e17b82 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:57.632479 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1770b1aa-07b5-46cf-b6b1-79ab078b38dc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:57.633957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1770b1aa-07b5-46cf-b6b1-79ab078b38dc None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:57.667943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09efee1f-4a09-4c7f-98e0-2cf421af5e9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:57.668609 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09efee1f-4a09-4c7f-98e0-2cf421af5e9a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:57.668881 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09efee1f-4a09-4c7f-98e0-2cf421af5e9a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:57.669146 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09efee1f-4a09-4c7f-98e0-2cf421af5e9a None admin] PATH_INFO: `/v3/projects/ada79d28652f4a3aa0a174133f24108f` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:57.671102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09efee1f-4a09-4c7f-98e0-2cf421af5e9a None admin] RBAC: Authorizing `identity:get_project(project_id=ada79d28652f4a3aa0a174133f24108f)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:57.672575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09efee1f-4a09-4c7f-98e0-2cf421af5e9a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:57.674234 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 230/448] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:57 2022] GET /identity/v3/projects/ada79d28652f4a3aa0a174133f24108f => generated 344 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:57.675478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09efee1f-4a09-4c7f-98e0-2cf421af5e9a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:57.685174 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:57.741853 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 219/449] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:04:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:04:57.742770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1770b1aa-07b5-46cf-b6b1-79ab078b38dc None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:57.748798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:57.784979 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09efee1f-4a09-4c7f-98e0-2cf421af5e9a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:57.821639 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66267525-1d41-4722-9d80-b8b1a9cf7f12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:57.822266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66267525-1d41-4722-9d80-b8b1a9cf7f12 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:57.822419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66267525-1d41-4722-9d80-b8b1a9cf7f12 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:57.822682 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66267525-1d41-4722-9d80-b8b1a9cf7f12 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:57.823406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66267525-1d41-4722-9d80-b8b1a9cf7f12 None admin] RBAC: Adding query filter params (group.id=9ba79133e99b4e91b58e94e32f101d4d, role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe, scope.project.id=ea5d616ff88c481baff44c2b3840d3d4) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:57.823703 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66267525-1d41-4722-9d80-b8b1a9cf7f12 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:57.825182 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66267525-1d41-4722-9d80-b8b1a9cf7f12 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:57.831285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 231/450] 10.222.0.47 () {56 vars in 1376 bytes} [Sun Aug 21 01:04:57 2022] GET /identity/v3/role_assignments?group.id=9ba79133e99b4e91b58e94e32f101d4d&scope.project.id=ea5d616ff88c481baff44c2b3840d3d4&role.id=9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 611 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:57.832123 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66267525-1d41-4722-9d80-b8b1a9cf7f12 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:57.864130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1770b1aa-07b5-46cf-b6b1-79ab078b38dc None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:57.907165 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33534040-a42a-431c-a214-4e3c8311baa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:57.908255 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33534040-a42a-431c-a214-4e3c8311baa2 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:57.908724 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33534040-a42a-431c-a214-4e3c8311baa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:57.908983 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33534040-a42a-431c-a214-4e3c8311baa2 None admin] PATH_INFO: `/v3/groups/22c4cfab6749435b9afdacfff854dd85` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:57.910266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33534040-a42a-431c-a214-4e3c8311baa2 None admin] RBAC: Authorizing `identity:get_group(group_id=22c4cfab6749435b9afdacfff854dd85)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:57.913266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33534040-a42a-431c-a214-4e3c8311baa2 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:57.915318 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 220/451] 10.222.0.47 () {56 vars in 1180 bytes} [Sun Aug 21 01:04:57 2022] GET /identity/v3/groups/22c4cfab6749435b9afdacfff854dd85 => generated 229 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:57.921235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33534040-a42a-431c-a214-4e3c8311baa2 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:57.942003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:57.962217 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66267525-1d41-4722-9d80-b8b1a9cf7f12 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:58.001274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:58.006574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a358240b-2901-4bcc-886d-baa6982899ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:58.007679 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a358240b-2901-4bcc-886d-baa6982899ac None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:58.007679 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a358240b-2901-4bcc-886d-baa6982899ac None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:58.007866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a358240b-2901-4bcc-886d-baa6982899ac None admin] PATH_INFO: `/v3/projects/ea5d616ff88c481baff44c2b3840d3d4` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:58.010736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a358240b-2901-4bcc-886d-baa6982899ac None admin] RBAC: Authorizing `identity:get_project(project_id=ea5d616ff88c481baff44c2b3840d3d4)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:58.010736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a358240b-2901-4bcc-886d-baa6982899ac None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:58.013469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 232/452] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:04:57 2022] GET /identity/v3/projects/ea5d616ff88c481baff44c2b3840d3d4 => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:04:58.019852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a358240b-2901-4bcc-886d-baa6982899ac None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:58.045886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33534040-a42a-431c-a214-4e3c8311baa2 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:58.088182 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f9a554-10dd-4e99-8fc7-2fd941777c70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:58.088732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f9a554-10dd-4e99-8fc7-2fd941777c70 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:58.089015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f9a554-10dd-4e99-8fc7-2fd941777c70 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:58.089286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f9a554-10dd-4e99-8fc7-2fd941777c70 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:58.090021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f9a554-10dd-4e99-8fc7-2fd941777c70 None admin] RBAC: Adding query filter params (group.id=22c4cfab6749435b9afdacfff854dd85, role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=ada79d28652f4a3aa0a174133f24108f) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:58.090332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f9a554-10dd-4e99-8fc7-2fd941777c70 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:58.091765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f9a554-10dd-4e99-8fc7-2fd941777c70 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:58.098212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 221/453] 10.222.0.47 () {56 vars in 1376 bytes} [Sun Aug 21 01:04:57 2022] GET /identity/v3/role_assignments?group.id=22c4cfab6749435b9afdacfff854dd85&scope.project.id=ada79d28652f4a3aa0a174133f24108f&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 611 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:58.155433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a358240b-2901-4bcc-886d-baa6982899ac None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:58.199267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79502985-82f7-417b-9309-586287473aad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:58.200410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79502985-82f7-417b-9309-586287473aad None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:58.200891 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79502985-82f7-417b-9309-586287473aad None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:58.201354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79502985-82f7-417b-9309-586287473aad None admin] PATH_INFO: `/v3/groups/9ba79133e99b4e91b58e94e32f101d4d` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:58.202877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79502985-82f7-417b-9309-586287473aad None admin] RBAC: Authorizing `identity:get_group(group_id=9ba79133e99b4e91b58e94e32f101d4d)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:58.204458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79502985-82f7-417b-9309-586287473aad None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:58.206345 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 233/454] 10.222.0.47 () {56 vars in 1180 bytes} [Sun Aug 21 01:04:58 2022] GET /identity/v3/groups/9ba79133e99b4e91b58e94e32f101d4d => generated 226 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:04:58.212977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f9a554-10dd-4e99-8fc7-2fd941777c70 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:04:58.306872 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:04:58.326893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f9a554-10dd-4e99-8fc7-2fd941777c70 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:04:58.362487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e25779-ef69-4620-a4d3-d80713ca4625 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:04:58.363098 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e25779-ef69-4620-a4d3-d80713ca4625 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:04:58.363376 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e25779-ef69-4620-a4d3-d80713ca4625 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:04:58.363650 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e25779-ef69-4620-a4d3-d80713ca4625 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:04:58.364403 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e25779-ef69-4620-a4d3-d80713ca4625 None admin] RBAC: Adding query filter params (group.id=9ba79133e99b4e91b58e94e32f101d4d, role.id=fbc5961470b74a85bde6be2171f5f08f, scope.project.id=ea5d616ff88c481baff44c2b3840d3d4) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:04:58.364707 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e25779-ef69-4620-a4d3-d80713ca4625 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:04:58.366149 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e25779-ef69-4620-a4d3-d80713ca4625 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:04:58.372669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 222/455] 10.222.0.47 () {56 vars in 1376 bytes} [Sun Aug 21 01:04:58 2022] GET /identity/v3/role_assignments?group.id=9ba79133e99b4e91b58e94e32f101d4d&scope.project.id=ea5d616ff88c481baff44c2b3840d3d4&role.id=fbc5961470b74a85bde6be2171f5f08f => generated 611 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:00.820122 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d907e812-8a40-43b3-bd06-d612c8171b3e None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:00.820657 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d907e812-8a40-43b3-bd06-d612c8171b3e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:00.820657 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d907e812-8a40-43b3-bd06-d612c8171b3e None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:00.822400 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 234/456] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:00.827906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ab836d-3c3b-4eb2-a398-e695e2f4cd52 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:00.828846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ab836d-3c3b-4eb2-a398-e695e2f4cd52 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:00.829337 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ab836d-3c3b-4eb2-a398-e695e2f4cd52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:00.857209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-62ab836d-3c3b-4eb2-a398-e695e2f4cd52 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:00.858796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-62ab836d-3c3b-4eb2-a398-e695e2f4cd52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:00.942259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 223/457] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:01.188667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa67d12e-ce46-4fad-afbb-4b4cf6ffdb2f None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:01.189271 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa67d12e-ce46-4fad-afbb-4b4cf6ffdb2f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:01.189756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa67d12e-ce46-4fad-afbb-4b4cf6ffdb2f None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:01.191606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 235/458] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:01.197396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483673c4-2382-46b7-ba9c-920c84241ce4 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:01.197924 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483673c4-2382-46b7-ba9c-920c84241ce4 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:01.198394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483673c4-2382-46b7-ba9c-920c84241ce4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:01.229115 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-483673c4-2382-46b7-ba9c-920c84241ce4 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:01.230326 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-483673c4-2382-46b7-ba9c-920c84241ce4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:01.306072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa67d12e-ce46-4fad-afbb-4b4cf6ffdb2f None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:01.319308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 224/459] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:01.331600 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214c2380-96a9-4ec4-b7af-1fad4697b73c None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:01.331885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214c2380-96a9-4ec4-b7af-1fad4697b73c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:01.332192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214c2380-96a9-4ec4-b7af-1fad4697b73c None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:01.333794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 225/460] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:05:01.338945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ca2b24-1ec3-4c87-86b2-1f75c6844de9 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:01.339230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ca2b24-1ec3-4c87-86b2-1f75c6844de9 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:01.344537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ca2b24-1ec3-4c87-86b2-1f75c6844de9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:01.376704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:01.382196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f8ca2b24-1ec3-4c87-86b2-1f75c6844de9 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:01.384710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f8ca2b24-1ec3-4c87-86b2-1f75c6844de9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:01.387567 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:01.428441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa67d12e-ce46-4fad-afbb-4b4cf6ffdb2f None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:01.484887 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80a86a6-53e5-487b-b5fc-d93e3369d578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:01.485529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80a86a6-53e5-487b-b5fc-d93e3369d578 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:01.485799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80a86a6-53e5-487b-b5fc-d93e3369d578 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:01.486067 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80a86a6-53e5-487b-b5fc-d93e3369d578 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:01.499554 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80a86a6-53e5-487b-b5fc-d93e3369d578 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:01.501448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80a86a6-53e5-487b-b5fc-d93e3369d578 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:01.514222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 226/461] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Aug 21 01:05:01.527818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-b80a86a6-53e5-487b-b5fc-d93e3369d578 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:01.527818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:01.527818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:01.527818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:01.527818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:01.527818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:01.527818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:01.527818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:01.527818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:01.527818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:01.527818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:01.527818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:01.527818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:01.527818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:01.527818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:01.527818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:01.528772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:01.529625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:01.529625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:01.529625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:01.536240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 236/462] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:01.538665 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ca2b24-1ec3-4c87-86b2-1f75c6844de9 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:01.660196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80a86a6-53e5-487b-b5fc-d93e3369d578 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:01.682106 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ca2b24-1ec3-4c87-86b2-1f75c6844de9 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:01.720271 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4006ef4-5661-4dfd-bc72-c920125bf49c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:01.721152 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4006ef4-5661-4dfd-bc72-c920125bf49c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:01.721605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4006ef4-5661-4dfd-bc72-c920125bf49c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:01.722076 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4006ef4-5661-4dfd-bc72-c920125bf49c None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:01.722906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4006ef4-5661-4dfd-bc72-c920125bf49c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:01.723395 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4006ef4-5661-4dfd-bc72-c920125bf49c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:01.725528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4006ef4-5661-4dfd-bc72-c920125bf49c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:01.727184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:01.744351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 227/463] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:01.744351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af620ab8-d797-48a6-90e1-df572405e1c3 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:01.744351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af620ab8-d797-48a6-90e1-df572405e1c3 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:01.744351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af620ab8-d797-48a6-90e1-df572405e1c3 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:01.746378 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 228/464] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:01 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:01.747421 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af620ab8-d797-48a6-90e1-df572405e1c3 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:01.792914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80a86a6-53e5-487b-b5fc-d93e3369d578 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:01.810313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:01.830182 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8956975-5fef-43dd-9fbb-4ca60b2559fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:01.830828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8956975-5fef-43dd-9fbb-4ca60b2559fa None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:01.831021 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8956975-5fef-43dd-9fbb-4ca60b2559fa None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:01.831201 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8956975-5fef-43dd-9fbb-4ca60b2559fa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:01.843398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8956975-5fef-43dd-9fbb-4ca60b2559fa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:01.845186 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8956975-5fef-43dd-9fbb-4ca60b2559fa None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:01.859619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af620ab8-d797-48a6-90e1-df572405e1c3 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:01.862759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-a8956975-5fef-43dd-9fbb-4ca60b2559fa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:01.862759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:01.862759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:01.862759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:01.862759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:01.862759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:01.862759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:01.862759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:01.862759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:01.862759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:01.862759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:01.862759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:01.862759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:01.862759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:01.862759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:01.862759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:01.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:01.864960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:01.864960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:01.864960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:01.871726 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 237/465] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:01.873342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af364b38-cdd7-4cf9-8e73-f70ca9fd58f6 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:01.873771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af364b38-cdd7-4cf9-8e73-f70ca9fd58f6 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:01.874189 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af364b38-cdd7-4cf9-8e73-f70ca9fd58f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:01.894038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-af364b38-cdd7-4cf9-8e73-f70ca9fd58f6 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:01.895543 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-af364b38-cdd7-4cf9-8e73-f70ca9fd58f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:01.899990 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad40159a-3b4f-41de-b624-2a6e89671648 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:01.900777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad40159a-3b4f-41de-b624-2a6e89671648 None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:01.901065 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad40159a-3b4f-41de-b624-2a6e89671648 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:01.901335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad40159a-3b4f-41de-b624-2a6e89671648 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:01.901976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad40159a-3b4f-41de-b624-2a6e89671648 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:01.903449 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad40159a-3b4f-41de-b624-2a6e89671648 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:01.970033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 238/466] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:01.975389 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af364b38-cdd7-4cf9-8e73-f70ca9fd58f6 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:02.034513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 229/467] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:05:01 2022] POST /identity/v3/users => generated 246 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:05:02.034513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad40159a-3b4f-41de-b624-2a6e89671648 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:02.134374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:02.148278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:02.152804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af364b38-cdd7-4cf9-8e73-f70ca9fd58f6 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:02.224053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afd4fc4-bcca-4455-805a-20025ab8e51d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:02.225698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afd4fc4-bcca-4455-805a-20025ab8e51d None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:02.226195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afd4fc4-bcca-4455-805a-20025ab8e51d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:02.226689 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afd4fc4-bcca-4455-805a-20025ab8e51d None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:02.228372 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afd4fc4-bcca-4455-805a-20025ab8e51d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:02.229061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afd4fc4-bcca-4455-805a-20025ab8e51d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:02.231197 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad40159a-3b4f-41de-b624-2a6e89671648 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:02.233045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afd4fc4-bcca-4455-805a-20025ab8e51d None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:02.262590 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 239/468] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:02.262590 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afd4fc4-bcca-4455-805a-20025ab8e51d None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:02.304003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e193bf66-3894-450d-a13e-d834cfb03307 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:02.304003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e193bf66-3894-450d-a13e-d834cfb03307 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:02.304003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e193bf66-3894-450d-a13e-d834cfb03307 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:02.304003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e193bf66-3894-450d-a13e-d834cfb03307 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:02.307690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e193bf66-3894-450d-a13e-d834cfb03307 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:02.309256 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e193bf66-3894-450d-a13e-d834cfb03307 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:02.329281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-e193bf66-3894-450d-a13e-d834cfb03307 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:02.329281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:02.329281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:02.329281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:02.329281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:02.329281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:02.329281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:02.329281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:02.329281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:02.329281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:02.329281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:02.329281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:02.329281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:02.329281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:02.329281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:02.329281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:02.330294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:02.331136 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:02.331136 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:02.331136 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:02.331136 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 230/469] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:02.332062 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e193bf66-3894-450d-a13e-d834cfb03307 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:02.402893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afd4fc4-bcca-4455-805a-20025ab8e51d None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:02.454657 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700eacb8-35c3-40a5-8de4-009be774f8ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:02.454860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e193bf66-3894-450d-a13e-d834cfb03307 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:02.455387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700eacb8-35c3-40a5-8de4-009be774f8ee None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:02.455775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700eacb8-35c3-40a5-8de4-009be774f8ee None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:02.456130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700eacb8-35c3-40a5-8de4-009be774f8ee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:02.457460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:02.471510 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700eacb8-35c3-40a5-8de4-009be774f8ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:02.473841 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700eacb8-35c3-40a5-8de4-009be774f8ee None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:02.494371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a67ba6-5f81-47ed-b399-c4fc295ee297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:02.494577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-700eacb8-35c3-40a5-8de4-009be774f8ee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:02.494577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:02.494577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:02.494577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:02.494577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:02.494577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:02.494577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:02.494577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:02.494577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:02.494577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:02.494577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:02.494577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:02.494577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:02.494577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:02.494577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:02.494577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:02.495428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:02.496788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:02.496788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:02.496788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:02.496957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a67ba6-5f81-47ed-b399-c4fc295ee297 None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:02.496957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a67ba6-5f81-47ed-b399-c4fc295ee297 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:02.496957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a67ba6-5f81-47ed-b399-c4fc295ee297 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:02.496957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a67ba6-5f81-47ed-b399-c4fc295ee297 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:02.497658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a67ba6-5f81-47ed-b399-c4fc295ee297 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:02.499770 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 240/470] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:02.501035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700eacb8-35c3-40a5-8de4-009be774f8ee None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:02.523227 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:02.564563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 231/471] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:05:02 2022] POST /identity/v3/users => generated 248 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:05:02.566197 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f5ae97-4b64-46e9-95b6-7e1ea70284d3 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:02.566617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f5ae97-4b64-46e9-95b6-7e1ea70284d3 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:02.567079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f5ae97-4b64-46e9-95b6-7e1ea70284d3 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:02.569353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 232/472] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:02.570553 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f5ae97-4b64-46e9-95b6-7e1ea70284d3 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:02.636898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700eacb8-35c3-40a5-8de4-009be774f8ee None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:02.678265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8ab639-4465-48ed-8a08-8a313e4f3fa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:02.679197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8ab639-4465-48ed-8a08-8a313e4f3fa8 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:02.679612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8ab639-4465-48ed-8a08-8a313e4f3fa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:02.680119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8ab639-4465-48ed-8a08-8a313e4f3fa8 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:02.680923 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8ab639-4465-48ed-8a08-8a313e4f3fa8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:02.681373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8ab639-4465-48ed-8a08-8a313e4f3fa8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:02.683022 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8ab639-4465-48ed-8a08-8a313e4f3fa8 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:02.698675 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 241/473] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:02.700365 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6669a347-7e07-4add-beab-db6d96deabac None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:02.700809 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6669a347-7e07-4add-beab-db6d96deabac None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:02.701231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6669a347-7e07-4add-beab-db6d96deabac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:02.710150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f5ae97-4b64-46e9-95b6-7e1ea70284d3 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:02.721263 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-6669a347-7e07-4add-beab-db6d96deabac None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:02.722752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6669a347-7e07-4add-beab-db6d96deabac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:02.757097 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f54cc52-c702-48d0-b00f-a49916a52c81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:02.759697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f54cc52-c702-48d0-b00f-a49916a52c81 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:02.759697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f54cc52-c702-48d0-b00f-a49916a52c81 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:02.759697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f54cc52-c702-48d0-b00f-a49916a52c81 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:02.759697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f54cc52-c702-48d0-b00f-a49916a52c81 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:02.759697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f54cc52-c702-48d0-b00f-a49916a52c81 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:02.760898 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f54cc52-c702-48d0-b00f-a49916a52c81 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:02.777106 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 233/474] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:02.777916 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f54cc52-c702-48d0-b00f-a49916a52c81 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:02.802623 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:02.816656 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 242/475] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:02.817748 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6669a347-7e07-4add-beab-db6d96deabac None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:02.823495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:02.901651 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f54cc52-c702-48d0-b00f-a49916a52c81 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:02.939433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475e9e5c-6a03-494a-aaa0-406a2c49e45f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:02.940045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475e9e5c-6a03-494a-aaa0-406a2c49e45f None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:02.940347 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475e9e5c-6a03-494a-aaa0-406a2c49e45f None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:02.940609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475e9e5c-6a03-494a-aaa0-406a2c49e45f None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:02.941239 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475e9e5c-6a03-494a-aaa0-406a2c49e45f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:02.942640 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475e9e5c-6a03-494a-aaa0-406a2c49e45f None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:02.968089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6669a347-7e07-4add-beab-db6d96deabac None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:03.007443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e61597e-aece-4b83-9448-e4754fcf1fff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:03.008052 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e61597e-aece-4b83-9448-e4754fcf1fff None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:03.008486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e61597e-aece-4b83-9448-e4754fcf1fff None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:03.008802 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e61597e-aece-4b83-9448-e4754fcf1fff None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:03.009441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e61597e-aece-4b83-9448-e4754fcf1fff None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:03.010904 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e61597e-aece-4b83-9448-e4754fcf1fff None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:03.044949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 234/476] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:05:02 2022] POST /identity/v3/users => generated 248 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:05:03.119508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 243/477] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:05:02 2022] POST /identity/v3/users => generated 249 bytes in 302 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:05:03.166855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475e9e5c-6a03-494a-aaa0-406a2c49e45f None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:03.257892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:03.285961 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475e9e5c-6a03-494a-aaa0-406a2c49e45f None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:03.326318 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc63d39c-effa-4862-8c98-c5f78cbd16e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:03.328269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc63d39c-effa-4862-8c98-c5f78cbd16e6 None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:03.328269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc63d39c-effa-4862-8c98-c5f78cbd16e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:03.328269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc63d39c-effa-4862-8c98-c5f78cbd16e6 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:03.328269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc63d39c-effa-4862-8c98-c5f78cbd16e6 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:03.332128 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc63d39c-effa-4862-8c98-c5f78cbd16e6 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:03.367626 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.sql.core [None req-cc63d39c-effa-4862-8c98-c5f78cbd16e6 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 21 01:05:03.367626 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 21 01:05:03.367626 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [parameters: {'id': '2d5698a4f2d047b784d20624d44fe88c', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 21 01:05:03.367626 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81805) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 21 01:05:03.385232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-cc63d39c-effa-4862-8c98-c5f78cbd16e6 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 21 01:05:03.385232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:03.385232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 21 01:05:03.385232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 21 01:05:03.385232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 21 01:05:03.385232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 21 01:05:03.385232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 21 01:05:03.385232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application result = self._query(query) Aug 21 01:05:03.385232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 21 01:05:03.385232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application conn.query(q) Aug 21 01:05:03.385232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 21 01:05:03.385232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 21 01:05:03.385232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 21 01:05:03.385232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application result.read() Aug 21 01:05:03.385232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return ref.to_dict() Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 21 01:05:03.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application next(self.gen) Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application yield resource Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application next(self.gen) Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self.session.rollback() Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self.force_reraise() Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise self.value Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application session.commit() Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 21 01:05:03.387139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._prepare_impl() Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self.session.flush() Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._flush(objects) Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application compat.raise_( Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application flush_context.execute() Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rec.execute(self) Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 21 01:05:03.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 21 01:05:03.391122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 21 01:05:03.391122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 21 01:05:03.391122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 21 01:05:03.391122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 21 01:05:03.391122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 21 01:05:03.391122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 21 01:05:03.391122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 21 01:05:03.391122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 21 01:05:03.391122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 21 01:05:03.391122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 21 01:05:03.391122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 21 01:05:03.391122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 21 01:05:03.391122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 21 01:05:03.391122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 21 01:05:03.391122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 21 01:05:03.391122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception Aug 21 01:05:03.393696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 21 01:05:03.393696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 21 01:05:03.393696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 21 01:05:03.393696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 21 01:05:03.393696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 21 01:05:03.393696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application result = self._query(query) Aug 21 01:05:03.393696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 21 01:05:03.393696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application conn.query(q) Aug 21 01:05:03.393696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 21 01:05:03.393696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 21 01:05:03.393696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 21 01:05:03.393696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application result.read() Aug 21 01:05:03.393696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 21 01:05:03.393696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 21 01:05:03.393696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 21 01:05:03.393696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 21 01:05:03.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 21 01:05:03.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 21 01:05:03.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 21 01:05:03.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 21 01:05:03.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 21 01:05:03.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 21 01:05:03.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [parameters: {'id': '2d5698a4f2d047b784d20624d44fe88c', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 21 01:05:03.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 21 01:05:03.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:03.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:03.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:03.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:03.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:03.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:03.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:03.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:03.397943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:03.397943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:03.397943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:03.397943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:03.397943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:03.397943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:03.397943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Aug 21 01:05:03.397943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Aug 21 01:05:03.397943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:03.397943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:03.397943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Aug 21 01:05:03.397943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Aug 21 01:05:03.397943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 21 01:05:03.397943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 21 01:05:03.397943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 21 01:05:03.397943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:03.400044 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 235/478] 10.222.0.47 () {60 vars in 1131 bytes} [Sun Aug 21 01:05:03 2022] POST /identity/v3/roles => generated 144 bytes in 220 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:03.400280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e61597e-aece-4b83-9448-e4754fcf1fff None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:03.462987 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:03.513360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e61597e-aece-4b83-9448-e4754fcf1fff None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:03.557598 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629a0911-8a3c-48d4-9ecd-8a04b4a7af78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:03.558232 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629a0911-8a3c-48d4-9ecd-8a04b4a7af78 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:03.558508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629a0911-8a3c-48d4-9ecd-8a04b4a7af78 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:03.558775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629a0911-8a3c-48d4-9ecd-8a04b4a7af78 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:03.573224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629a0911-8a3c-48d4-9ecd-8a04b4a7af78 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:03.578528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-629a0911-8a3c-48d4-9ecd-8a04b4a7af78 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:05:03.578528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:03.578528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:03.578528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:03.578528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:03.578528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:03.578528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:03.578528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:03.578528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:03.578528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:03.578528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:03.578528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:03.578528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:03.578528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:03.578528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:03.578528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:03.579402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:03.579402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:03.579402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:03.579402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:03.579402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:03.579402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:03.579402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:03.579402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:03.579402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:03.579402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:03.579402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:03.579402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:03.579402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:03.579402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:03.579402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:03.579402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:03.580838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:03.580838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:03.580838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:03.580838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:03.580838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:03.580838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:03.580838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:03.580838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:03.580838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:03.580838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:03.580838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 21 01:05:03.580838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:03.580838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 244/479] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:05:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:03.589166 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc63d39c-effa-4862-8c98-c5f78cbd16e6 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:03.733734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc63d39c-effa-4862-8c98-c5f78cbd16e6 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:03.775153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9097a93e-2ac2-48ce-9d88-059f49a25c2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:03.775769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9097a93e-2ac2-48ce-9d88-059f49a25c2b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:03.776043 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9097a93e-2ac2-48ce-9d88-059f49a25c2b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:03.776335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9097a93e-2ac2-48ce-9d88-059f49a25c2b None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:03.777000 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9097a93e-2ac2-48ce-9d88-059f49a25c2b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:03.777295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9097a93e-2ac2-48ce-9d88-059f49a25c2b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:03.778646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9097a93e-2ac2-48ce-9d88-059f49a25c2b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:03.784368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:03.789465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 236/480] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:03 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:04.415011 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a41b16-1a53-45bb-ab30-b2b8894f0d91 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:04.415514 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a41b16-1a53-45bb-ab30-b2b8894f0d91 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:04.415587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a41b16-1a53-45bb-ab30-b2b8894f0d91 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:04.417162 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 245/481] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:04.422140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d9472b-fe4f-4b7b-af6c-49f85c96eeb6 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:04.422547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d9472b-fe4f-4b7b-af6c-49f85c96eeb6 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:04.422721 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d9472b-fe4f-4b7b-af6c-49f85c96eeb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:04.444977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-35d9472b-fe4f-4b7b-af6c-49f85c96eeb6 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:04.446159 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-35d9472b-fe4f-4b7b-af6c-49f85c96eeb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:04.539975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 237/482] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:04.881758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b22cada-262a-49bb-8511-1c583734e8a4 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:04.882149 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b22cada-262a-49bb-8511-1c583734e8a4 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:04.882335 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b22cada-262a-49bb-8511-1c583734e8a4 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:04.883854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 246/483] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:04.889310 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e016618c-ca2c-4ef0-83b3-04a936176128 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:04.889596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e016618c-ca2c-4ef0-83b3-04a936176128 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:04.889865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e016618c-ca2c-4ef0-83b3-04a936176128 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:04.908626 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-e016618c-ca2c-4ef0-83b3-04a936176128 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:04.909813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e016618c-ca2c-4ef0-83b3-04a936176128 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:04.915682 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b22cada-262a-49bb-8511-1c583734e8a4 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:05.048476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 238/484] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:05:05.088840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b22cada-262a-49bb-8511-1c583734e8a4 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:05.114431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:05.139559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678e2732-f452-4daf-99bf-a4fbe148504b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:05.139559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678e2732-f452-4daf-99bf-a4fbe148504b None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:05.139559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678e2732-f452-4daf-99bf-a4fbe148504b None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:05.139559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678e2732-f452-4daf-99bf-a4fbe148504b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:05.156353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678e2732-f452-4daf-99bf-a4fbe148504b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:05.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-678e2732-f452-4daf-99bf-a4fbe148504b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:05.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:05.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:05.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:05.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:05.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:05.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:05.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:05.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:05.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:05.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:05.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:05.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:05.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:05.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:05.161434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:05.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:05.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:05.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:05.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:05.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:05.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:05.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:05.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:05.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:05.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:05.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:05.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:05.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:05.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:05.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:05.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:05.163141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:05.163141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:05.163141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:05.163141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:05.163141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:05.163141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:05.163141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:05.163141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:05.163141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:05.163141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:05.163141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:05.163141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:05.165771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 247/485] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:05.171144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e016618c-ca2c-4ef0-83b3-04a936176128 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:05.266920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:05.320974 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e016618c-ca2c-4ef0-83b3-04a936176128 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:05.389865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4190badc-12a4-48f8-8614-15087292ad18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:05.390411 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4190badc-12a4-48f8-8614-15087292ad18 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:05.390687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4190badc-12a4-48f8-8614-15087292ad18 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:05.390954 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4190badc-12a4-48f8-8614-15087292ad18 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:05.391680 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4190badc-12a4-48f8-8614-15087292ad18 None admin] RBAC: Adding query filter params (name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:05.391979 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4190badc-12a4-48f8-8614-15087292ad18 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:05.393361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4190badc-12a4-48f8-8614-15087292ad18 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:05.408734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 239/486] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:05.414563 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678e2732-f452-4daf-99bf-a4fbe148504b None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:05.512276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4190badc-12a4-48f8-8614-15087292ad18 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:05.579785 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678e2732-f452-4daf-99bf-a4fbe148504b None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:05.623120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde4a50d-1c02-43fc-a8e8-34681d5d7ed1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:05.624385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde4a50d-1c02-43fc-a8e8-34681d5d7ed1 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:05.624876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde4a50d-1c02-43fc-a8e8-34681d5d7ed1 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:05.625400 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde4a50d-1c02-43fc-a8e8-34681d5d7ed1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:05.637275 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde4a50d-1c02-43fc-a8e8-34681d5d7ed1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:05.638717 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde4a50d-1c02-43fc-a8e8-34681d5d7ed1 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:05.644494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4190badc-12a4-48f8-8614-15087292ad18 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:05.653492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-dde4a50d-1c02-43fc-a8e8-34681d5d7ed1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:05.653492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:05.653492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:05.653492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:05.653492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:05.653492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:05.653492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:05.653492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:05.653492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:05.653492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:05.653492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:05.653492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:05.653492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:05.653492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:05.653492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:05.653492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:05.654411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:05.655269 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:05.655269 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:05.655269 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:05.655269 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 248/487] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:05.656157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef0224-bfcf-4301-9968-708998e62cbf None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:05.656438 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef0224-bfcf-4301-9968-708998e62cbf None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:05.656705 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef0224-bfcf-4301-9968-708998e62cbf None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:05.658063 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 249/488] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:05 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:05.660792 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5f7657-4bf4-4487-aea5-d68435f0c285 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:05.661074 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5f7657-4bf4-4487-aea5-d68435f0c285 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:05.661344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5f7657-4bf4-4487-aea5-d68435f0c285 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:05.662604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 250/489] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:05.663319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5f7657-4bf4-4487-aea5-d68435f0c285 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:05.680829 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:05.694248 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40c4cef-d83e-43a2-bfc4-8b94b3f85584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:05.694879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40c4cef-d83e-43a2-bfc4-8b94b3f85584 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:05.695157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40c4cef-d83e-43a2-bfc4-8b94b3f85584 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:05.695420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40c4cef-d83e-43a2-bfc4-8b94b3f85584 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:05.700203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:05.706100 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40c4cef-d83e-43a2-bfc4-8b94b3f85584 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:05.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-e40c4cef-d83e-43a2-bfc4-8b94b3f85584 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:05.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:05.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:05.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:05.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:05.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:05.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:05.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:05.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:05.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:05.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:05.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:05.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:05.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:05.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:05.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:05.711927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:05.711927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:05.711927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:05.711927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:05.711927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:05.711927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:05.711927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:05.711927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:05.711927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:05.711927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:05.711927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:05.711927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:05.711927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:05.711927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:05.711927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:05.711927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:05.712952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:05.712952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:05.712952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:05.712952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:05.712952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:05.712952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:05.712952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:05.712952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:05.712952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:05.712952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:05.712952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:05.712952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:05.716247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 240/490] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:05.718569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7710c9d-46a9-4d28-b48d-dfd08722d9cd None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:05.718854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7710c9d-46a9-4d28-b48d-dfd08722d9cd None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:05.719398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7710c9d-46a9-4d28-b48d-dfd08722d9cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:05.738599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-a7710c9d-46a9-4d28-b48d-dfd08722d9cd None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:05.740353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a7710c9d-46a9-4d28-b48d-dfd08722d9cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:05.779411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5f7657-4bf4-4487-aea5-d68435f0c285 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:05.814120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 241/491] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:05.814120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93f6425-a31d-456f-a9ef-bfe4f605ce09 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:05.814120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93f6425-a31d-456f-a9ef-bfe4f605ce09 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:05.814120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93f6425-a31d-456f-a9ef-bfe4f605ce09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:05.826767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89b0f90-8185-4cb3-b57b-0108e6c65845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:05.827462 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f93f6425-a31d-456f-a9ef-bfe4f605ce09 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:05.827579 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89b0f90-8185-4cb3-b57b-0108e6c65845 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:05.827896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89b0f90-8185-4cb3-b57b-0108e6c65845 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:05.828313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89b0f90-8185-4cb3-b57b-0108e6c65845 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:05.828933 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f93f6425-a31d-456f-a9ef-bfe4f605ce09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:05.829187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89b0f90-8185-4cb3-b57b-0108e6c65845 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:05.829562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89b0f90-8185-4cb3-b57b-0108e6c65845 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:05.831145 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89b0f90-8185-4cb3-b57b-0108e6c65845 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:05.845225 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 251/492] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:05.846483 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89b0f90-8185-4cb3-b57b-0108e6c65845 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:05.912549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 242/493] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:05.913331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93f6425-a31d-456f-a9ef-bfe4f605ce09 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:05.961355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:06.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:06.028128 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93f6425-a31d-456f-a9ef-bfe4f605ce09 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:06.050048 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89b0f90-8185-4cb3-b57b-0108e6c65845 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:06.068593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2595b3c6-5062-482f-b93c-ef14328cd25a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:06.069455 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2595b3c6-5062-482f-b93c-ef14328cd25a None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:06.069877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2595b3c6-5062-482f-b93c-ef14328cd25a None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:06.070287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2595b3c6-5062-482f-b93c-ef14328cd25a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:06.078456 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2595b3c6-5062-482f-b93c-ef14328cd25a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:06.080078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2595b3c6-5062-482f-b93c-ef14328cd25a None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:06.095976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-2595b3c6-5062-482f-b93c-ef14328cd25a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:05:06.095976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:06.095976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:06.095976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:06.095976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:06.095976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:06.095976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:06.095976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:06.095976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:06.095976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:06.095976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:06.095976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:06.095976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:06.095976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:06.095976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:06.095976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:06.097146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:06.098359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:06.098359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:06.098359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:06.098359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:06.098359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:06.098359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:06.098359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:06.098359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:06.098359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:05:06.098359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:06.101303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 243/494] 10.222.0.47 () {56 vars in 1093 bytes} [Sun Aug 21 01:05:05 2022] GET /identity/v3/users/nova => generated 82 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:06.104891 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2595b3c6-5062-482f-b93c-ef14328cd25a None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:06.124072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e93ea97-0816-4136-8dac-01d2791e0707 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:06.124838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e93ea97-0816-4136-8dac-01d2791e0707 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:06.125110 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e93ea97-0816-4136-8dac-01d2791e0707 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:06.125377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e93ea97-0816-4136-8dac-01d2791e0707 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:06.126042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e93ea97-0816-4136-8dac-01d2791e0707 None admin] RBAC: Adding query filter params (name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:06.126339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e93ea97-0816-4136-8dac-01d2791e0707 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:06.127722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e93ea97-0816-4136-8dac-01d2791e0707 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:06.136639 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 252/495] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:06.137452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e93ea97-0816-4136-8dac-01d2791e0707 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:06.248744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e93ea97-0816-4136-8dac-01d2791e0707 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:06.250731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2595b3c6-5062-482f-b93c-ef14328cd25a None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:06.253400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:06.283573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6905d724-07ea-4e0c-ae56-725c01c12b07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:06.284198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6905d724-07ea-4e0c-ae56-725c01c12b07 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:06.284473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6905d724-07ea-4e0c-ae56-725c01c12b07 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:06.284733 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6905d724-07ea-4e0c-ae56-725c01c12b07 None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:06.285536 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6905d724-07ea-4e0c-ae56-725c01c12b07 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:06.285842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6905d724-07ea-4e0c-ae56-725c01c12b07 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:06.287928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6905d724-07ea-4e0c-ae56-725c01c12b07 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:06.288053 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa059f8-6b92-42fc-9f3b-538fa061bc19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:06.288557 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa059f8-6b92-42fc-9f3b-538fa061bc19 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:06.289109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa059f8-6b92-42fc-9f3b-538fa061bc19 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:06.289608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa059f8-6b92-42fc-9f3b-538fa061bc19 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:06.300245 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa059f8-6b92-42fc-9f3b-538fa061bc19 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:06.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-5fa059f8-6b92-42fc-9f3b-538fa061bc19 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:06.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:06.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:06.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:06.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:06.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:06.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:06.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:06.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:06.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:06.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:06.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:06.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:06.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:06.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:06.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:06.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:06.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:06.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:06.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:06.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:06.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:06.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:06.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:06.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:06.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:06.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:06.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:06.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:06.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:06.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:06.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:06.306687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:06.306687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:06.306687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:06.306687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:06.306687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:06.306687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:06.306687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:06.306687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:06.306687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:06.306687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:06.306687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:06.306687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:06.306687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 244/496] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:06.307350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa059f8-6b92-42fc-9f3b-538fa061bc19 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:06.324386 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 253/497] 10.222.0.47 () {56 vars in 1133 bytes} [Sun Aug 21 01:05:06 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:06.325130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6905d724-07ea-4e0c-ae56-725c01c12b07 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:06.337476 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:06.409089 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa059f8-6b92-42fc-9f3b-538fa061bc19 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:06.425181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6905d724-07ea-4e0c-ae56-725c01c12b07 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:06.442935 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcefac3-05de-484e-80a7-96bf81b573c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:06.443571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcefac3-05de-484e-80a7-96bf81b573c7 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:06.443845 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcefac3-05de-484e-80a7-96bf81b573c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:06.444123 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcefac3-05de-484e-80a7-96bf81b573c7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:06.454061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcefac3-05de-484e-80a7-96bf81b573c7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:06.455543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcefac3-05de-484e-80a7-96bf81b573c7 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:06.458121 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e20ca8-234d-4056-8d0b-d25d6b16267d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:06.459258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e20ca8-234d-4056-8d0b-d25d6b16267d None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:06.459530 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e20ca8-234d-4056-8d0b-d25d6b16267d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:06.459791 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e20ca8-234d-4056-8d0b-d25d6b16267d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:06.470575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e20ca8-234d-4056-8d0b-d25d6b16267d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:06.471559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-fdcefac3-05de-484e-80a7-96bf81b573c7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:06.471559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:06.471559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:06.471559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:06.471559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:06.471559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:06.471559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:06.471559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:06.471559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:06.471559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:06.471559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:06.471559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:06.471559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:06.471559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:06.471559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:06.471559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:06.472517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:06.473357 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:06.473357 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:06.473357 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:06.473357 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 245/498] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:06.473833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a0cad-a9be-4131-bb48-d62465c7242a None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a0cad-a9be-4131-bb48-d62465c7242a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:06.474991 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a0cad-a9be-4131-bb48-d62465c7242a None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:06.475899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-b0e20ca8-234d-4056-8d0b-d25d6b16267d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:06.475899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:06.475899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:06.475899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:06.475899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:06.475899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:06.475899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:06.475899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:06.475899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:06.475899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:06.475899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:06.475899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:06.475899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:06.475899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:06.475899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:06.475899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:06.481049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:06.481049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:06.481049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:06.481049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:06.481049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:06.481049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:06.481049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:06.481049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:06.481049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:06.481049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:06.481049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:06.481049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:06.481049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:06.481049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:06.481049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:06.481049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:06.482167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:06.482167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:06.482167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:06.482167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:06.482167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:06.482167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:06.482167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:06.482167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:06.482167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:06.482167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:06.482167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:06.482167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:06.483203 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 246/499] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:06.483203 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838a0cad-a9be-4131-bb48-d62465c7242a None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:06.483372 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 254/500] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:06.483372 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e20ca8-234d-4056-8d0b-d25d6b16267d None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:06.496210 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:06.552929 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:06.581940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838a0cad-a9be-4131-bb48-d62465c7242a None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:06.582763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e20ca8-234d-4056-8d0b-d25d6b16267d None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:06.616892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04b8014-5520-4c7c-89ba-76c5c4b4cf92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:06.617490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04b8014-5520-4c7c-89ba-76c5c4b4cf92 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:06.617768 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04b8014-5520-4c7c-89ba-76c5c4b4cf92 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:06.618039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04b8014-5520-4c7c-89ba-76c5c4b4cf92 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:06.618350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb9773e-8274-41df-886c-7be568057556 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:06.618690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04b8014-5520-4c7c-89ba-76c5c4b4cf92 None admin] RBAC: Adding query filter params (name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:06.618947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb9773e-8274-41df-886c-7be568057556 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:06.619033 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04b8014-5520-4c7c-89ba-76c5c4b4cf92 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:06.619223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb9773e-8274-41df-886c-7be568057556 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:06.619486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb9773e-8274-41df-886c-7be568057556 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:06.620366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04b8014-5520-4c7c-89ba-76c5c4b4cf92 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:06.628198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 247/501] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:06.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04b8014-5520-4c7c-89ba-76c5c4b4cf92 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:06.686940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb9773e-8274-41df-886c-7be568057556 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:06.688445 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb9773e-8274-41df-886c-7be568057556 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:06.707702 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:06.730013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04b8014-5520-4c7c-89ba-76c5c4b4cf92 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:06.756495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:06.763634 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba974f9-76e0-4377-87be-773080ddac6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:06.764275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba974f9-76e0-4377-87be-773080ddac6d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:06.764552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba974f9-76e0-4377-87be-773080ddac6d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:06.764822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba974f9-76e0-4377-87be-773080ddac6d None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:06.765446 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba974f9-76e0-4377-87be-773080ddac6d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:06.765743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba974f9-76e0-4377-87be-773080ddac6d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:06.767100 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba974f9-76e0-4377-87be-773080ddac6d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:06.778723 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 248/502] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:06.779893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb75762-1b21-4319-9123-5ce03b6c8825 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:06.780199 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb75762-1b21-4319-9123-5ce03b6c8825 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:06.780481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb75762-1b21-4319-9123-5ce03b6c8825 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:06.789517 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 255/503] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:05:06.790425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb9773e-8274-41df-886c-7be568057556 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:06.799074 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-fdb75762-1b21-4319-9123-5ce03b6c8825 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:06.800299 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-fdb75762-1b21-4319-9123-5ce03b6c8825 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:06.867364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 249/504] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:06.868131 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb75762-1b21-4319-9123-5ce03b6c8825 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:06.890639 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb9773e-8274-41df-886c-7be568057556 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:06.927155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013c248e-fe8b-44c2-8fee-4203dfc3e45e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:06.928293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013c248e-fe8b-44c2-8fee-4203dfc3e45e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:06.928566 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013c248e-fe8b-44c2-8fee-4203dfc3e45e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:06.928837 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013c248e-fe8b-44c2-8fee-4203dfc3e45e None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:06.929496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013c248e-fe8b-44c2-8fee-4203dfc3e45e None admin] RBAC: Adding query filter params (name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:06.929795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013c248e-fe8b-44c2-8fee-4203dfc3e45e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:06.931167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013c248e-fe8b-44c2-8fee-4203dfc3e45e None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:06.944284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 256/505] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:06.944284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013c248e-fe8b-44c2-8fee-4203dfc3e45e None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:06.945672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:06.976026 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:06.981396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb75762-1b21-4319-9123-5ce03b6c8825 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:07.016499 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ca07dc-5655-4cb2-accc-62a1bd07d6ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:07.017312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ca07dc-5655-4cb2-accc-62a1bd07d6ca None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:07.017590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ca07dc-5655-4cb2-accc-62a1bd07d6ca None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:07.017861 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ca07dc-5655-4cb2-accc-62a1bd07d6ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:07.028770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ca07dc-5655-4cb2-accc-62a1bd07d6ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:07.030210 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ca07dc-5655-4cb2-accc-62a1bd07d6ca None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:07.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-25ca07dc-5655-4cb2-accc-62a1bd07d6ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:07.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:07.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:07.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:07.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:07.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:07.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:07.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:07.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:07.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:07.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:07.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:07.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:07.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:07.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:07.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:07.046989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:07.047846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:07.047846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:07.047846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:07.047846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 250/506] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:07.048071 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ca07dc-5655-4cb2-accc-62a1bd07d6ca None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:07.050648 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013c248e-fe8b-44c2-8fee-4203dfc3e45e None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:07.090231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7a650f-2f27-48fa-92a6-7c449a584588 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:07.090849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a650f-2f27-48fa-92a6-7c449a584588 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:07.091119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a650f-2f27-48fa-92a6-7c449a584588 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:07.091384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a650f-2f27-48fa-92a6-7c449a584588 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:07.098952 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7a650f-2f27-48fa-92a6-7c449a584588 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:07.100889 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7a650f-2f27-48fa-92a6-7c449a584588 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:07.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-cf7a650f-2f27-48fa-92a6-7c449a584588 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 21 01:05:07.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:07.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:07.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:07.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:07.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:07.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:07.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:07.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:07.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:07.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:07.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:07.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:07.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:07.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:07.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:07.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:07.116440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:07.116440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:07.116440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:07.116440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:07.116440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:07.116440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:07.116440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:07.116440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:07.116440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 21 01:05:07.116440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:07.116440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 257/507] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:05:06 2022] GET /identity/v3/users/cinder => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:07.116440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7a650f-2f27-48fa-92a6-7c449a584588 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:07.163210 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ca07dc-5655-4cb2-accc-62a1bd07d6ca None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:07.180078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:07.197268 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8994d4-eea7-49a4-8021-6c98a04e860c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:07.197987 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8994d4-eea7-49a4-8021-6c98a04e860c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:07.198262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8994d4-eea7-49a4-8021-6c98a04e860c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:07.198530 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8994d4-eea7-49a4-8021-6c98a04e860c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:07.208914 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8994d4-eea7-49a4-8021-6c98a04e860c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:07.210354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8994d4-eea7-49a4-8021-6c98a04e860c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:07.218177 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:07.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-ac8994d4-eea7-49a4-8021-6c98a04e860c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:07.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:07.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:07.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:07.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:07.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:07.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:07.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:07.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:07.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:07.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:07.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:07.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:07.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:07.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:07.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:07.225986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:07.227025 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:07.227025 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:07.227025 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:07.227232 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7a650f-2f27-48fa-92a6-7c449a584588 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:07.228480 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 251/508] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:07.229498 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8994d4-eea7-49a4-8021-6c98a04e860c None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:07.261864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e5043e-1b76-4f9f-8085-047bfafc0754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:07.262487 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e5043e-1b76-4f9f-8085-047bfafc0754 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:07.262812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e5043e-1b76-4f9f-8085-047bfafc0754 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:07.263082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e5043e-1b76-4f9f-8085-047bfafc0754 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:07.273437 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e5043e-1b76-4f9f-8085-047bfafc0754 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:07.274879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e5043e-1b76-4f9f-8085-047bfafc0754 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:07.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-97e5043e-1b76-4f9f-8085-047bfafc0754 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:07.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:07.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:07.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:07.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:07.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:07.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:07.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:07.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:07.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:07.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:07.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:07.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:07.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:07.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:07.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:07.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:07.291009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:07.291009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:07.291009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:07.291009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 258/509] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:07.295215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e5043e-1b76-4f9f-8085-047bfafc0754 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:07.333564 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8994d4-eea7-49a4-8021-6c98a04e860c None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:07.368189 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a0ccf9-778a-4059-96d1-cbaa245539e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:07.368790 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a0ccf9-778a-4059-96d1-cbaa245539e4 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:07.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a0ccf9-778a-4059-96d1-cbaa245539e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:07.369385 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a0ccf9-778a-4059-96d1-cbaa245539e4 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:07.370011 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a0ccf9-778a-4059-96d1-cbaa245539e4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:07.370308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a0ccf9-778a-4059-96d1-cbaa245539e4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:07.371673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a0ccf9-778a-4059-96d1-cbaa245539e4 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:07.385000 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 252/510] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:07.385832 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a0ccf9-778a-4059-96d1-cbaa245539e4 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:07.398659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e5043e-1b76-4f9f-8085-047bfafc0754 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:07.416730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:07.433564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d7c10e-5062-4a70-b1a6-53a9dd872545 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:07.434434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d7c10e-5062-4a70-b1a6-53a9dd872545 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:07.434810 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d7c10e-5062-4a70-b1a6-53a9dd872545 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:07.435167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d7c10e-5062-4a70-b1a6-53a9dd872545 None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:07.436265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d7c10e-5062-4a70-b1a6-53a9dd872545 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:07.436722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d7c10e-5062-4a70-b1a6-53a9dd872545 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:07.438274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d7c10e-5062-4a70-b1a6-53a9dd872545 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:07.456411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 259/511] 10.222.0.47 () {56 vars in 1137 bytes} [Sun Aug 21 01:05:07 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:07.457416 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d7c10e-5062-4a70-b1a6-53a9dd872545 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:07.460261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:07.489580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a0ccf9-778a-4059-96d1-cbaa245539e4 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:07.524239 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8bda4d-a5d3-488d-8402-be2f48b8d635 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:07.525162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8bda4d-a5d3-488d-8402-be2f48b8d635 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:07.525446 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8bda4d-a5d3-488d-8402-be2f48b8d635 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:07.525720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8bda4d-a5d3-488d-8402-be2f48b8d635 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:07.538037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8bda4d-a5d3-488d-8402-be2f48b8d635 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:07.539485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8bda4d-a5d3-488d-8402-be2f48b8d635 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:07.554415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-3b8bda4d-a5d3-488d-8402-be2f48b8d635 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:07.554415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:07.554415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:07.554415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:07.554415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:07.554415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:07.554415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:07.554415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:07.554415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:07.554415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:07.554415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:07.554415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:07.554415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:07.554415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:07.554415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:07.554415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:07.555388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:07.556296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:07.556296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:07.556296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:07.556296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 253/512] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:07.556296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8bda4d-a5d3-488d-8402-be2f48b8d635 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:07.563953 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d7c10e-5062-4a70-b1a6-53a9dd872545 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:07.597930 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b861c57-b8d9-4b58-a107-bf15d752337a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:07.598546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b861c57-b8d9-4b58-a107-bf15d752337a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:07.599313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b861c57-b8d9-4b58-a107-bf15d752337a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:07.599571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b861c57-b8d9-4b58-a107-bf15d752337a None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:07.600218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b861c57-b8d9-4b58-a107-bf15d752337a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:07.600513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b861c57-b8d9-4b58-a107-bf15d752337a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:07.601873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b861c57-b8d9-4b58-a107-bf15d752337a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:07.612799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 260/513] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:07.613568 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b861c57-b8d9-4b58-a107-bf15d752337a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:07.638314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:07.656610 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8bda4d-a5d3-488d-8402-be2f48b8d635 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:07.675300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:07.693092 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19a74c2-ebe2-49ab-b376-f15b18ae882d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:07.693697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19a74c2-ebe2-49ab-b376-f15b18ae882d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:07.693977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19a74c2-ebe2-49ab-b376-f15b18ae882d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:07.694251 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19a74c2-ebe2-49ab-b376-f15b18ae882d None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:07.694925 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19a74c2-ebe2-49ab-b376-f15b18ae882d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:07.695224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19a74c2-ebe2-49ab-b376-f15b18ae882d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:07.696608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19a74c2-ebe2-49ab-b376-f15b18ae882d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:07.708643 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 254/514] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:07.709247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19a74c2-ebe2-49ab-b376-f15b18ae882d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:07.718650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b861c57-b8d9-4b58-a107-bf15d752337a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:07.753975 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bbfe07-951e-4ce6-a576-f8eeb3cc4402 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:07.754604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bbfe07-951e-4ce6-a576-f8eeb3cc4402 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:07.754878 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bbfe07-951e-4ce6-a576-f8eeb3cc4402 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:07.755144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bbfe07-951e-4ce6-a576-f8eeb3cc4402 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:07.762517 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bbfe07-951e-4ce6-a576-f8eeb3cc4402 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:07.763951 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bbfe07-951e-4ce6-a576-f8eeb3cc4402 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:07.775507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-c7bbfe07-951e-4ce6-a576-f8eeb3cc4402 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 21 01:05:07.775507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:07.775507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:07.775507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:07.775507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:07.775507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:07.775507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:07.775507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:07.775507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:07.775507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:07.775507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:07.775507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:07.775507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:07.775507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:07.775507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:07.775507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:07.776434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:07.777328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:07.777328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:07.777328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:07.777328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:07.777328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:07.777328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:07.777328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:07.777328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:07.777328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 21 01:05:07.777328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:07.777328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 261/515] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:07 2022] GET /identity/v3/users/neutron => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:07.777885 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bbfe07-951e-4ce6-a576-f8eeb3cc4402 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:07.812628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19a74c2-ebe2-49ab-b376-f15b18ae882d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:07.846690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b79722-dcb0-4a3a-9cc8-16e680db0ecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:07.847373 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b79722-dcb0-4a3a-9cc8-16e680db0ecf None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:07.847659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b79722-dcb0-4a3a-9cc8-16e680db0ecf None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:07.847933 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b79722-dcb0-4a3a-9cc8-16e680db0ecf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:07.860138 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:07.877672 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bbfe07-951e-4ce6-a576-f8eeb3cc4402 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:07.900214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:07.912830 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bf9627-c504-4b1f-b415-3e644b5e4b66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:07.913433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bf9627-c504-4b1f-b415-3e644b5e4b66 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:07.913710 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bf9627-c504-4b1f-b415-3e644b5e4b66 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:07.913984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bf9627-c504-4b1f-b415-3e644b5e4b66 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:07.914621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bf9627-c504-4b1f-b415-3e644b5e4b66 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:07.914924 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bf9627-c504-4b1f-b415-3e644b5e4b66 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:07.916317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bf9627-c504-4b1f-b415-3e644b5e4b66 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:07.919068 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b79722-dcb0-4a3a-9cc8-16e680db0ecf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:07.920638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b79722-dcb0-4a3a-9cc8-16e680db0ecf None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:07.928152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 262/516] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:07.928898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bf9627-c504-4b1f-b415-3e644b5e4b66 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:08.022894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 255/517] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:05:08.023841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b79722-dcb0-4a3a-9cc8-16e680db0ecf None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:08.033421 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bf9627-c504-4b1f-b415-3e644b5e4b66 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:08.065346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:08.067499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc089ad-2982-44e4-bd47-7102ec1ff573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:08.068543 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc089ad-2982-44e4-bd47-7102ec1ff573 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:08.068816 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc089ad-2982-44e4-bd47-7102ec1ff573 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:08.069078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc089ad-2982-44e4-bd47-7102ec1ff573 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:08.075680 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc089ad-2982-44e4-bd47-7102ec1ff573 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:08.077134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc089ad-2982-44e4-bd47-7102ec1ff573 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:08.089084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-abc089ad-2982-44e4-bd47-7102ec1ff573 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:08.089084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:08.089084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:08.089084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:08.089084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:08.089084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:08.089084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:08.089084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:08.089084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:08.089084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:08.089084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:08.089084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:08.089084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:08.089084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:08.089084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:08.090139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:08.095996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:08.095996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:08.095996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:08.095996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:08.095996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:08.095996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:08.095996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:08.095996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:08.095996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 263/518] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:08.095996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc089ad-2982-44e4-bd47-7102ec1ff573 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:08.122750 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b79722-dcb0-4a3a-9cc8-16e680db0ecf None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:08.135728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:08.157835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4690012-25a2-45c1-9888-b7de6a151d54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:08.158453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4690012-25a2-45c1-9888-b7de6a151d54 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:08.158729 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4690012-25a2-45c1-9888-b7de6a151d54 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:08.158993 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4690012-25a2-45c1-9888-b7de6a151d54 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:08.165901 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4690012-25a2-45c1-9888-b7de6a151d54 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:08.167337 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4690012-25a2-45c1-9888-b7de6a151d54 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:08.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-c4690012-25a2-45c1-9888-b7de6a151d54 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 21 01:05:08.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:08.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:08.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:08.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:08.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:08.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:08.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:08.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:08.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:08.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:08.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:08.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:08.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:08.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:08.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:08.180388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:08.181302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:08.181302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:08.181302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:08.181302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:08.181302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:08.181302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:08.181302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:08.181302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:08.181302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 21 01:05:08.181302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:08.181302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 256/519] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:05:08 2022] GET /identity/v3/users/glance => generated 84 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:08.181854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4690012-25a2-45c1-9888-b7de6a151d54 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:08.195454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc089ad-2982-44e4-bd47-7102ec1ff573 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:08.229340 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2764c39f-54b6-4277-891d-ae847c8e1c04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:08.229941 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2764c39f-54b6-4277-891d-ae847c8e1c04 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:08.230216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2764c39f-54b6-4277-891d-ae847c8e1c04 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:08.230485 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2764c39f-54b6-4277-891d-ae847c8e1c04 None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:08.231199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2764c39f-54b6-4277-891d-ae847c8e1c04 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:08.231499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2764c39f-54b6-4277-891d-ae847c8e1c04 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:08.232973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2764c39f-54b6-4277-891d-ae847c8e1c04 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:08.250300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 264/520] 10.222.0.47 () {56 vars in 1139 bytes} [Sun Aug 21 01:05:08 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:08.251085 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2764c39f-54b6-4277-891d-ae847c8e1c04 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:08.283803 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4690012-25a2-45c1-9888-b7de6a151d54 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:08.285361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:08.320548 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a36ef4-c7e4-4598-bab1-b59b66b54b92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:08.321355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a36ef4-c7e4-4598-bab1-b59b66b54b92 None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:08.321823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a36ef4-c7e4-4598-bab1-b59b66b54b92 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:08.322401 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a36ef4-c7e4-4598-bab1-b59b66b54b92 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:08.323101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a36ef4-c7e4-4598-bab1-b59b66b54b92 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:08.324552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a36ef4-c7e4-4598-bab1-b59b66b54b92 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:08.350254 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:08.353149 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2764c39f-54b6-4277-891d-ae847c8e1c04 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:08.386340 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353dc82b-fdb0-48cb-94b7-a3a7b72737ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:08.386962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353dc82b-fdb0-48cb-94b7-a3a7b72737ec None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:08.387237 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353dc82b-fdb0-48cb-94b7-a3a7b72737ec None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:08.387505 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353dc82b-fdb0-48cb-94b7-a3a7b72737ec None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:08.397888 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353dc82b-fdb0-48cb-94b7-a3a7b72737ec None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:08.399326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353dc82b-fdb0-48cb-94b7-a3a7b72737ec None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:08.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-353dc82b-fdb0-48cb-94b7-a3a7b72737ec None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:08.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:08.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:08.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:08.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:08.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:08.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:08.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:08.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:08.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:08.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:08.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:08.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:08.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:08.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:08.413439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:08.414527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:08.415375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:08.415375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:08.415375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:08.415375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 265/521] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:08.415375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353dc82b-fdb0-48cb-94b7-a3a7b72737ec None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:08.448821 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 257/522] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:05:08 2022] POST /identity/v3/users => generated 247 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:05:08.449568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a36ef4-c7e4-4598-bab1-b59b66b54b92 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:08.527758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353dc82b-fdb0-48cb-94b7-a3a7b72737ec None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:08.564036 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea56488a-5f80-4900-8fb4-e015dbe6dfd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:08.564682 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea56488a-5f80-4900-8fb4-e015dbe6dfd9 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:08.564958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea56488a-5f80-4900-8fb4-e015dbe6dfd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:08.565231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea56488a-5f80-4900-8fb4-e015dbe6dfd9 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:08.565927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea56488a-5f80-4900-8fb4-e015dbe6dfd9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:08.566221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea56488a-5f80-4900-8fb4-e015dbe6dfd9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:08.568461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea56488a-5f80-4900-8fb4-e015dbe6dfd9 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:08.579647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a36ef4-c7e4-4598-bab1-b59b66b54b92 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:08.586983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 266/523] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:08.588075 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea56488a-5f80-4900-8fb4-e015dbe6dfd9 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:08.605599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:08.621751 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e91e91-fc5d-4a1f-963f-4655b3ef7e8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:08.622981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e91e91-fc5d-4a1f-963f-4655b3ef7e8c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:08.622981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e91e91-fc5d-4a1f-963f-4655b3ef7e8c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:08.623167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e91e91-fc5d-4a1f-963f-4655b3ef7e8c None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:08.624134 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e91e91-fc5d-4a1f-963f-4655b3ef7e8c None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:08.624134 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e91e91-fc5d-4a1f-963f-4655b3ef7e8c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:08.625541 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e91e91-fc5d-4a1f-963f-4655b3ef7e8c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:08.666602 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 258/524] 10.222.0.47 () {56 vars in 1137 bytes} [Sun Aug 21 01:05:08 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:08.668168 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e91e91-fc5d-4a1f-963f-4655b3ef7e8c None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:08.687158 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:08.707127 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea56488a-5f80-4900-8fb4-e015dbe6dfd9 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:08.743768 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74efb56b-0af5-41ce-bd68-ae7273a038f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:08.744661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74efb56b-0af5-41ce-bd68-ae7273a038f2 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:08.744941 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74efb56b-0af5-41ce-bd68-ae7273a038f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:08.745268 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74efb56b-0af5-41ce-bd68-ae7273a038f2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:08.785196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e91e91-fc5d-4a1f-963f-4655b3ef7e8c None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:08.820181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74efb56b-0af5-41ce-bd68-ae7273a038f2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:08.821649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74efb56b-0af5-41ce-bd68-ae7273a038f2 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:08.821751 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbac75d4-f951-482c-83ab-21f0af47809c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:08.822253 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbac75d4-f951-482c-83ab-21f0af47809c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:08.822526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbac75d4-f951-482c-83ab-21f0af47809c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:08.822794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbac75d4-f951-482c-83ab-21f0af47809c None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:08.823433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbac75d4-f951-482c-83ab-21f0af47809c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:08.823729 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbac75d4-f951-482c-83ab-21f0af47809c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:08.825136 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbac75d4-f951-482c-83ab-21f0af47809c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:08.837621 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 259/525] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:08.838431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbac75d4-f951-482c-83ab-21f0af47809c None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:08.842998 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:08.920963 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:08.943958 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbac75d4-f951-482c-83ab-21f0af47809c None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:08.945100 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 267/526] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:05:08.946410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74efb56b-0af5-41ce-bd68-ae7273a038f2 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:08.984262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7236246b-3c2e-4997-8acc-df16e6e60e0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:08.985666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7236246b-3c2e-4997-8acc-df16e6e60e0e None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:08.985956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7236246b-3c2e-4997-8acc-df16e6e60e0e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:08.986225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7236246b-3c2e-4997-8acc-df16e6e60e0e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:08.986946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7236246b-3c2e-4997-8acc-df16e6e60e0e None admin] RBAC: Adding query filter params (role.id=c2d8bf7283e14857879d03d1f39c0165, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=fabdcdc25d1d4ef6bc66bd6cadecaaf8) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:08.987250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7236246b-3c2e-4997-8acc-df16e6e60e0e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:08.988731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7236246b-3c2e-4997-8acc-df16e6e60e0e None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:08.996281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 260/527] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:08 2022] GET /identity/v3/role_assignments?user.id=fabdcdc25d1d4ef6bc66bd6cadecaaf8&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=c2d8bf7283e14857879d03d1f39c0165 => generated 259 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:08.996954 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7236246b-3c2e-4997-8acc-df16e6e60e0e None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:09.070995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74efb56b-0af5-41ce-bd68-ae7273a038f2 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:09.095215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:09.117777 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940eadc2-44a3-407e-8b45-87a4170e4458 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:09.118921 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940eadc2-44a3-407e-8b45-87a4170e4458 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:09.119952 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940eadc2-44a3-407e-8b45-87a4170e4458 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:09.120669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940eadc2-44a3-407e-8b45-87a4170e4458 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:09.143752 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7236246b-3c2e-4997-8acc-df16e6e60e0e None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:09.187083 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db094804-41b5-4296-b2d0-a70106fcc29b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:09.187718 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db094804-41b5-4296-b2d0-a70106fcc29b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:09.188021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db094804-41b5-4296-b2d0-a70106fcc29b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:09.188437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db094804-41b5-4296-b2d0-a70106fcc29b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:09.196463 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db094804-41b5-4296-b2d0-a70106fcc29b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:09.197949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db094804-41b5-4296-b2d0-a70106fcc29b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:09.200873 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:09.210856 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940eadc2-44a3-407e-8b45-87a4170e4458 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:09.212950 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940eadc2-44a3-407e-8b45-87a4170e4458 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:09.213059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-db094804-41b5-4296-b2d0-a70106fcc29b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:09.213059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:09.213059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:09.213059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:09.213059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:09.213059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:09.213059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:09.213059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:09.213059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:09.213059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:09.213059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:09.213059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:09.213059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:09.213059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:09.213059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:09.213975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:09.214965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:09.214965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:09.214965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:09.214965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:09.214965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:09.214965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:09.214965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:09.214965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:09.214965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 261/528] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:09.214965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db094804-41b5-4296-b2d0-a70106fcc29b None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:09.324868 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 268/529] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:05:09.326103 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940eadc2-44a3-407e-8b45-87a4170e4458 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:09.329104 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db094804-41b5-4296-b2d0-a70106fcc29b None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:09.342546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:09.367669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c070a22-98b2-482f-a819-222b4372ad22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:09.368331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c070a22-98b2-482f-a819-222b4372ad22 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:09.368613 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c070a22-98b2-482f-a819-222b4372ad22 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:09.368885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c070a22-98b2-482f-a819-222b4372ad22 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:09.380065 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c070a22-98b2-482f-a819-222b4372ad22 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:09.381609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c070a22-98b2-482f-a819-222b4372ad22 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:09.399031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-0c070a22-98b2-482f-a819-222b4372ad22 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:09.399031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:09.399031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:09.399031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:09.399031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:09.399031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:09.399031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:09.399031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:09.399031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:09.399031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:09.399031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:09.399031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:09.399031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:09.399031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:09.399031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:09.399031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:09.400205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:09.401090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:09.401090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:09.401090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:09.401090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 262/530] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:09.401090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c070a22-98b2-482f-a819-222b4372ad22 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:09.441482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940eadc2-44a3-407e-8b45-87a4170e4458 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:09.459793 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:09.480883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3a25f9-b0f3-43e2-87bd-e9f374237c8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:09.483284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3a25f9-b0f3-43e2-87bd-e9f374237c8e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:09.483284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3a25f9-b0f3-43e2-87bd-e9f374237c8e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:09.483284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3a25f9-b0f3-43e2-87bd-e9f374237c8e None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:09.483284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3a25f9-b0f3-43e2-87bd-e9f374237c8e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:09.483799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3a25f9-b0f3-43e2-87bd-e9f374237c8e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:09.485890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3a25f9-b0f3-43e2-87bd-e9f374237c8e None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:09.503123 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 269/531] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:09.504482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3a25f9-b0f3-43e2-87bd-e9f374237c8e None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:09.513867 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c070a22-98b2-482f-a819-222b4372ad22 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:09.554394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f8868e-7eba-480b-a4a5-15f841491e04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:09.555071 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f8868e-7eba-480b-a4a5-15f841491e04 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:09.555472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f8868e-7eba-480b-a4a5-15f841491e04 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:09.555748 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f8868e-7eba-480b-a4a5-15f841491e04 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:09.568507 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f8868e-7eba-480b-a4a5-15f841491e04 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:09.570228 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f8868e-7eba-480b-a4a5-15f841491e04 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:09.586334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-29f8868e-7eba-480b-a4a5-15f841491e04 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:09.586334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:09.586334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:09.586334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:09.586334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:09.586334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:09.586334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:09.586334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:09.586334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:09.586334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:09.586334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:09.586334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:09.586334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:09.586334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:09.586334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:09.586334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:09.587275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:09.588156 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:09.588156 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:09.588156 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:09.588156 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 263/532] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:09.588397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f8868e-7eba-480b-a4a5-15f841491e04 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:09.602761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:09.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3a25f9-b0f3-43e2-87bd-e9f374237c8e None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:09.683071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8033e3-dffc-4c00-9f5e-ff0c3219d913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:09.683817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8033e3-dffc-4c00-9f5e-ff0c3219d913 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:09.684223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8033e3-dffc-4c00-9f5e-ff0c3219d913 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:09.684618 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8033e3-dffc-4c00-9f5e-ff0c3219d913 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:09.685536 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8033e3-dffc-4c00-9f5e-ff0c3219d913 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:09.685934 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8033e3-dffc-4c00-9f5e-ff0c3219d913 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:09.687675 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8033e3-dffc-4c00-9f5e-ff0c3219d913 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:09.701703 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 270/533] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:09.702819 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8033e3-dffc-4c00-9f5e-ff0c3219d913 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:09.713562 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f8868e-7eba-480b-a4a5-15f841491e04 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:09.717595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:09.758607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c5d7f4-9538-4756-9920-44eb6434114d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:09.759437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c5d7f4-9538-4756-9920-44eb6434114d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:09.759858 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c5d7f4-9538-4756-9920-44eb6434114d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:09.760309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c5d7f4-9538-4756-9920-44eb6434114d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:09.761189 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c5d7f4-9538-4756-9920-44eb6434114d None admin] RBAC: Adding query filter params (role.id=c2d8bf7283e14857879d03d1f39c0165, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=b953036b8b774f25bda59cec96b21c27) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:09.762058 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c5d7f4-9538-4756-9920-44eb6434114d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:09.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c5d7f4-9538-4756-9920-44eb6434114d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:09.772144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 264/534] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:09 2022] GET /identity/v3/role_assignments?user.id=b953036b8b774f25bda59cec96b21c27&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=c2d8bf7283e14857879d03d1f39c0165 => generated 259 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:09.773653 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c5d7f4-9538-4756-9920-44eb6434114d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:09.822286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8033e3-dffc-4c00-9f5e-ff0c3219d913 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:09.877625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:09.888815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70983a48-041f-46dd-8e28-197fc7cf7678 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:09.889970 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70983a48-041f-46dd-8e28-197fc7cf7678 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:09.890168 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70983a48-041f-46dd-8e28-197fc7cf7678 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:09.890607 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70983a48-041f-46dd-8e28-197fc7cf7678 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:09.892193 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70983a48-041f-46dd-8e28-197fc7cf7678 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:09.892193 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70983a48-041f-46dd-8e28-197fc7cf7678 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:09.893675 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70983a48-041f-46dd-8e28-197fc7cf7678 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:09.912253 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c5d7f4-9538-4756-9920-44eb6434114d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:09.919401 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 271/535] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:09.924582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70983a48-041f-46dd-8e28-197fc7cf7678 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:09.959587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb3495c-072b-4da4-b63b-1756d58f8f3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:09.960637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb3495c-072b-4da4-b63b-1756d58f8f3b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:09.962741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb3495c-072b-4da4-b63b-1756d58f8f3b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:09.963324 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb3495c-072b-4da4-b63b-1756d58f8f3b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:09.978793 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb3495c-072b-4da4-b63b-1756d58f8f3b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:09.980304 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb3495c-072b-4da4-b63b-1756d58f8f3b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:09.993367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-adb3495c-072b-4da4-b63b-1756d58f8f3b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:09.993367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:09.993367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:09.993367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:09.993367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:09.993367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:09.993367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:09.993367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:09.993367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:09.993367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:09.993367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:09.993367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:09.993367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:09.993367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:09.993367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:09.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:09.995230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:09.995230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:09.995230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:09.995230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:09.995230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:09.995230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:09.995230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:09.995230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:09.995230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 265/536] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:10.000523 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb3495c-072b-4da4-b63b-1756d58f8f3b None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:10.070787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70983a48-041f-46dd-8e28-197fc7cf7678 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:10.077520 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:10.116914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd8c01b-7f76-4773-a037-1d95814125ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:10.117548 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd8c01b-7f76-4773-a037-1d95814125ef None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:10.117826 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd8c01b-7f76-4773-a037-1d95814125ef None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:10.118093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd8c01b-7f76-4773-a037-1d95814125ef None admin] PATH_INFO: `/v3/projects/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:10.134631 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd8c01b-7f76-4773-a037-1d95814125ef None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:10.136018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd8c01b-7f76-4773-a037-1d95814125ef None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:10.158171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-dcd8c01b-7f76-4773-a037-1d95814125ef None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:10.158171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:10.158171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:10.158171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:10.158171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:10.158171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:10.158171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:10.158171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:10.158171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:10.158171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:10.158171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:10.158171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:10.158171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:10.158171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:10.158171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:10.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:10.159939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:10.159939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:10.159939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:10.159939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:10.159939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:10.159939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:10.159939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:10.159939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:10.161584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 272/537] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:10.166559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd8c01b-7f76-4773-a037-1d95814125ef None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:10.220771 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb3495c-072b-4da4-b63b-1756d58f8f3b None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:10.258089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:10.284472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ec2ba4-f10b-4ea1-864c-ed88658fbddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:10.285078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ec2ba4-f10b-4ea1-864c-ed88658fbddd None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:10.285356 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ec2ba4-f10b-4ea1-864c-ed88658fbddd None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:10.285628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ec2ba4-f10b-4ea1-864c-ed88658fbddd None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:10.286508 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ec2ba4-f10b-4ea1-864c-ed88658fbddd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:10.286813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ec2ba4-f10b-4ea1-864c-ed88658fbddd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:10.288264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ec2ba4-f10b-4ea1-864c-ed88658fbddd None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:10.333573 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 266/538] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:10.334849 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ec2ba4-f10b-4ea1-864c-ed88658fbddd None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:10.355200 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd8c01b-7f76-4773-a037-1d95814125ef None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:10.416914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac6f531-93e4-4734-b1ee-20a334e5abba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:10.417448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac6f531-93e4-4734-b1ee-20a334e5abba None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:10.417722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac6f531-93e4-4734-b1ee-20a334e5abba None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:10.418053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac6f531-93e4-4734-b1ee-20a334e5abba None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:10.418763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac6f531-93e4-4734-b1ee-20a334e5abba None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:10.419059 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac6f531-93e4-4734-b1ee-20a334e5abba None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:10.420529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac6f531-93e4-4734-b1ee-20a334e5abba None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:10.447585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 273/539] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:10.452607 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac6f531-93e4-4734-b1ee-20a334e5abba None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:10.504547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:10.523578 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ec2ba4-f10b-4ea1-864c-ed88658fbddd None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:10.560510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92699360-1e5d-42b9-9158-a2c4d1b47c08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:10.561123 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92699360-1e5d-42b9-9158-a2c4d1b47c08 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:10.561771 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92699360-1e5d-42b9-9158-a2c4d1b47c08 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:10.562128 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92699360-1e5d-42b9-9158-a2c4d1b47c08 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:10.562928 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92699360-1e5d-42b9-9158-a2c4d1b47c08 None admin] RBAC: Adding query filter params (role.id=c2d8bf7283e14857879d03d1f39c0165, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=b532ddab1f5e467989edaa9e4b74bd72) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:10.563305 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92699360-1e5d-42b9-9158-a2c4d1b47c08 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:10.564844 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92699360-1e5d-42b9-9158-a2c4d1b47c08 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:10.576263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 267/540] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:10 2022] GET /identity/v3/role_assignments?user.id=b532ddab1f5e467989edaa9e4b74bd72&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=c2d8bf7283e14857879d03d1f39c0165 => generated 259 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:10.577842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d67bbe5-aefd-4242-8ead-20d80b95b44e None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:10.578198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d67bbe5-aefd-4242-8ead-20d80b95b44e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:10.578544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d67bbe5-aefd-4242-8ead-20d80b95b44e None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:10.579959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 268/541] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:10.586407 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365a59f-e796-458a-8930-1f0a7a534fc2 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:10.586688 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365a59f-e796-458a-8930-1f0a7a534fc2 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:10.587097 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365a59f-e796-458a-8930-1f0a7a534fc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:10.600555 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac6f531-93e4-4734-b1ee-20a334e5abba None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:10.618883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:10.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f365a59f-e796-458a-8930-1f0a7a534fc2 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:10.621568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f365a59f-e796-458a-8930-1f0a7a534fc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:10.650141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32513ad5-fb90-48dc-9e2c-50ed2df08238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:10.650802 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32513ad5-fb90-48dc-9e2c-50ed2df08238 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:10.651036 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32513ad5-fb90-48dc-9e2c-50ed2df08238 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:10.651311 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32513ad5-fb90-48dc-9e2c-50ed2df08238 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:10.653191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32513ad5-fb90-48dc-9e2c-50ed2df08238 None admin] RBAC: Adding query filter params (role.id=c2d8bf7283e14857879d03d1f39c0165, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=a6365b2328f34ab9b1b73ececbf5cc30) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:10.655694 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32513ad5-fb90-48dc-9e2c-50ed2df08238 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:10.658126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32513ad5-fb90-48dc-9e2c-50ed2df08238 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:10.670625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 274/542] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:10 2022] GET /identity/v3/role_assignments?user.id=a6365b2328f34ab9b1b73ececbf5cc30&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=c2d8bf7283e14857879d03d1f39c0165 => generated 259 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:10.731352 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 269/543] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:05:11.045834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32513ad5-fb90-48dc-9e2c-50ed2df08238 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:11.135973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e103a6b0-5c3a-4160-99c9-0aca6e5f661d None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:11.136443 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e103a6b0-5c3a-4160-99c9-0aca6e5f661d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:11.136916 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e103a6b0-5c3a-4160-99c9-0aca6e5f661d None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:11.139915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 270/544] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:11.144984 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616435b3-32b4-4034-831e-907836973c65 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:11.145419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616435b3-32b4-4034-831e-907836973c65 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:11.145953 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616435b3-32b4-4034-831e-907836973c65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:11.175122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-616435b3-32b4-4034-831e-907836973c65 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:11.176342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-616435b3-32b4-4034-831e-907836973c65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:11.187729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32513ad5-fb90-48dc-9e2c-50ed2df08238 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:11.224623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef4ff38-4ea4-48bc-80e0-f588fd633bf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:11.225260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef4ff38-4ea4-48bc-80e0-f588fd633bf1 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:11.225535 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef4ff38-4ea4-48bc-80e0-f588fd633bf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:11.225808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef4ff38-4ea4-48bc-80e0-f588fd633bf1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:11.236492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef4ff38-4ea4-48bc-80e0-f588fd633bf1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:11.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-cef4ff38-4ea4-48bc-80e0-f588fd633bf1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:11.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:11.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:11.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:11.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:11.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:11.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:11.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:11.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:11.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:11.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:11.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:11.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:11.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:11.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:11.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:11.242623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:11.242623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:11.242623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:11.242623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:11.242623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:11.242623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:11.242623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:11.242623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:11.242623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:11.242623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:11.242623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:11.242623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:11.242623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:11.242623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:11.242623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:11.242623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:11.243428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:11.243428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:11.243428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:11.243428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:11.243428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:11.243428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:11.243428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:11.243428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:11.243428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:11.243428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:11.243428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:11.243428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:11.243428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 275/545] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:11.254013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef4ff38-4ea4-48bc-80e0-f588fd633bf1 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:11.291989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:11.329456 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 271/546] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:05:11.370978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef4ff38-4ea4-48bc-80e0-f588fd633bf1 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:11.418705 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f064db3b-7026-4c33-8ee9-01d49ee4cfca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:11.419306 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f064db3b-7026-4c33-8ee9-01d49ee4cfca None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:11.419577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f064db3b-7026-4c33-8ee9-01d49ee4cfca None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:11.419852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f064db3b-7026-4c33-8ee9-01d49ee4cfca None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:11.420546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f064db3b-7026-4c33-8ee9-01d49ee4cfca None admin] RBAC: Adding query filter params (name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:11.420844 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f064db3b-7026-4c33-8ee9-01d49ee4cfca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:11.422219 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f064db3b-7026-4c33-8ee9-01d49ee4cfca None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:11.431216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 276/547] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:11 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:11.439653 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616435b3-32b4-4034-831e-907836973c65 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:11.456557 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:11.571549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616435b3-32b4-4034-831e-907836973c65 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:11.609190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4c26ca-ccd2-4b86-86dc-b60176ad2b13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:11.610154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4c26ca-ccd2-4b86-86dc-b60176ad2b13 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:11.610633 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4c26ca-ccd2-4b86-86dc-b60176ad2b13 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:11.611091 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4c26ca-ccd2-4b86-86dc-b60176ad2b13 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:11.626235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4c26ca-ccd2-4b86-86dc-b60176ad2b13 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:11.627922 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4c26ca-ccd2-4b86-86dc-b60176ad2b13 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:11.646909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-aa4c26ca-ccd2-4b86-86dc-b60176ad2b13 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:11.646909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:11.646909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:11.646909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:11.646909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:11.646909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:11.646909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:11.646909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:11.646909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:11.646909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:11.646909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:11.646909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:11.646909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:11.646909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:11.646909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:11.646909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:11.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:11.648823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:11.648823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:11.648823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:11.650609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 272/548] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:11.658874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f064db3b-7026-4c33-8ee9-01d49ee4cfca None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:11.739192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4c26ca-ccd2-4b86-86dc-b60176ad2b13 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:11.778070 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:11.784799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f064db3b-7026-4c33-8ee9-01d49ee4cfca None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:11.853258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bc9035-cbea-4a91-a935-2accd4f02ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:11.853866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bc9035-cbea-4a91-a935-2accd4f02ab4 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:11.854442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bc9035-cbea-4a91-a935-2accd4f02ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:11.854716 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bc9035-cbea-4a91-a935-2accd4f02ab4 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:11.857983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1bc9035-cbea-4a91-a935-2accd4f02ab4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:11.858291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1bc9035-cbea-4a91-a935-2accd4f02ab4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:11.859726 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1bc9035-cbea-4a91-a935-2accd4f02ab4 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:11.878050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 277/549] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:11.892137 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:11.892325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bc9035-cbea-4a91-a935-2accd4f02ab4 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:11.963938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4c26ca-ccd2-4b86-86dc-b60176ad2b13 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:12.025202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3ba434-cd2a-47e9-b4de-b6b5c3850617 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:12.025904 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3ba434-cd2a-47e9-b4de-b6b5c3850617 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:12.026189 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3ba434-cd2a-47e9-b4de-b6b5c3850617 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:12.026460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3ba434-cd2a-47e9-b4de-b6b5c3850617 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:12.038486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3ba434-cd2a-47e9-b4de-b6b5c3850617 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:12.040079 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bc9035-cbea-4a91-a935-2accd4f02ab4 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:12.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-4a3ba434-cd2a-47e9-b4de-b6b5c3850617 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:12.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:12.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:12.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:12.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:12.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:12.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:12.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:12.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:12.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:12.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:12.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:12.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:12.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:12.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:12.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:12.044732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:12.044732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:12.044732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:12.044732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:12.044732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:12.044732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:12.044732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:12.044732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:12.044732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:12.044732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:12.044732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:12.044732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:12.044732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:12.044732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:12.044732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:12.044732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:12.045543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:12.045543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:12.045543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:12.045543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:12.045543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:12.045543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:12.045543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:12.045543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:12.045543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:12.045543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:12.045543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:12.045543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:12.045543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 273/550] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:12.057516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3ba434-cd2a-47e9-b4de-b6b5c3850617 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:12.103524 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc6aa9b-ebda-4537-9ca6-4d91737fd922 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:12.104249 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc6aa9b-ebda-4537-9ca6-4d91737fd922 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:12.104530 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc6aa9b-ebda-4537-9ca6-4d91737fd922 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:12.104801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc6aa9b-ebda-4537-9ca6-4d91737fd922 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:12.114850 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc6aa9b-ebda-4537-9ca6-4d91737fd922 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:12.116938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc6aa9b-ebda-4537-9ca6-4d91737fd922 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:12.134399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-7dc6aa9b-ebda-4537-9ca6-4d91737fd922 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 21 01:05:12.134399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:12.134399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:12.134399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:12.134399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:12.134399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:12.134399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:12.134399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:12.134399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:12.134399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:12.134399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:12.134399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:12.134399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:12.134399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:12.134399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:12.134399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:12.135486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:12.136433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:12.136433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:12.136433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:12.136433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:12.136433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:12.136433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:12.136433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:12.136433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:12.136433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 21 01:05:12.136433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:12.136433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 278/551] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:05:11 2022] GET /identity/v3/users/swift => generated 83 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:12.137494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9522f66b-e249-42e2-94cb-6820490c581c None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:12.137494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9522f66b-e249-42e2-94cb-6820490c581c None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:12.137740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9522f66b-e249-42e2-94cb-6820490c581c None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:12.143961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 279/552] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:12.145105 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9522f66b-e249-42e2-94cb-6820490c581c None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:12.175533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:12.191351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3ba434-cd2a-47e9-b4de-b6b5c3850617 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:12.218133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:12.230988 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c8b50e-54f3-42e4-984b-c487b974c529 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:12.231692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c8b50e-54f3-42e4-984b-c487b974c529 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:12.231971 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c8b50e-54f3-42e4-984b-c487b974c529 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:12.232355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c8b50e-54f3-42e4-984b-c487b974c529 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:12.233067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c8b50e-54f3-42e4-984b-c487b974c529 None admin] RBAC: Adding query filter params (name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:12.233301 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c8b50e-54f3-42e4-984b-c487b974c529 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:12.234756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c8b50e-54f3-42e4-984b-c487b974c529 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:12.248373 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 274/553] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:12.249513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c058b6bb-c3e6-4a37-9eff-4f4673e0ca75 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:12.249791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c058b6bb-c3e6-4a37-9eff-4f4673e0ca75 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:12.250061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c058b6bb-c3e6-4a37-9eff-4f4673e0ca75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:12.271169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-c058b6bb-c3e6-4a37-9eff-4f4673e0ca75 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:12.272420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-c058b6bb-c3e6-4a37-9eff-4f4673e0ca75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:12.292524 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9522f66b-e249-42e2-94cb-6820490c581c None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:12.330699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96632644-6a64-4de4-8219-061d7f3d2d15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:12.331576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96632644-6a64-4de4-8219-061d7f3d2d15 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:12.331895 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96632644-6a64-4de4-8219-061d7f3d2d15 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:12.332227 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96632644-6a64-4de4-8219-061d7f3d2d15 None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:12.333290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96632644-6a64-4de4-8219-061d7f3d2d15 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:12.333642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96632644-6a64-4de4-8219-061d7f3d2d15 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:12.335075 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96632644-6a64-4de4-8219-061d7f3d2d15 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:12.368729 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 275/554] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:12.368729 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c058b6bb-c3e6-4a37-9eff-4f4673e0ca75 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:12.372439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 280/555] 10.222.0.47 () {56 vars in 1135 bytes} [Sun Aug 21 01:05:12 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:12.382467 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96632644-6a64-4de4-8219-061d7f3d2d15 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:12.511254 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:12.539224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96632644-6a64-4de4-8219-061d7f3d2d15 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:12.584124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c058b6bb-c3e6-4a37-9eff-4f4673e0ca75 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:12.589670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d3bd52-cde3-45db-a236-d0194a2e5905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:12.590283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d3bd52-cde3-45db-a236-d0194a2e5905 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:12.590560 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d3bd52-cde3-45db-a236-d0194a2e5905 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:12.590830 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d3bd52-cde3-45db-a236-d0194a2e5905 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:12.628724 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9aec77-aec5-4fa7-a1f5-9eea7444ec32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:12.629268 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9aec77-aec5-4fa7-a1f5-9eea7444ec32 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:12.629544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9aec77-aec5-4fa7-a1f5-9eea7444ec32 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:12.630377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9aec77-aec5-4fa7-a1f5-9eea7444ec32 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:12.640750 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9aec77-aec5-4fa7-a1f5-9eea7444ec32 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:12.642186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9aec77-aec5-4fa7-a1f5-9eea7444ec32 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:12.652457 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:12.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-bf9aec77-aec5-4fa7-a1f5-9eea7444ec32 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:12.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:12.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:12.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:12.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:12.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:12.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:12.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:12.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:12.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:12.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:12.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:12.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:12.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:12.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:12.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:12.663146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:12.664135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:12.664135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:12.664135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:12.664135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 276/556] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 298 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:12.670096 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9aec77-aec5-4fa7-a1f5-9eea7444ec32 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:12.728766 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d3bd52-cde3-45db-a236-d0194a2e5905 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:12.730262 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d3bd52-cde3-45db-a236-d0194a2e5905 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:12.799990 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9aec77-aec5-4fa7-a1f5-9eea7444ec32 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:12.827122 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:12.837427 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6ad2db-4256-423b-b41c-d5d75508d94c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:12.838205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6ad2db-4256-423b-b41c-d5d75508d94c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:12.838479 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6ad2db-4256-423b-b41c-d5d75508d94c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:12.838745 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6ad2db-4256-423b-b41c-d5d75508d94c None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:12.839381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6ad2db-4256-423b-b41c-d5d75508d94c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:12.839674 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6ad2db-4256-423b-b41c-d5d75508d94c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:12.841048 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6ad2db-4256-423b-b41c-d5d75508d94c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:12.854535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 277/557] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:12.856202 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 281/558] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 474 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 21 01:05:12.857082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d3bd52-cde3-45db-a236-d0194a2e5905 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:12.859812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b13a19-8003-44e8-930f-a16c3dd2ee24 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:12.860094 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b13a19-8003-44e8-930f-a16c3dd2ee24 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:12.860408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b13a19-8003-44e8-930f-a16c3dd2ee24 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:12.861643 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 278/559] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:12.862316 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b13a19-8003-44e8-930f-a16c3dd2ee24 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:12.891729 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:12.964218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d3bd52-cde3-45db-a236-d0194a2e5905 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:12.972918 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b13a19-8003-44e8-930f-a16c3dd2ee24 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:13.001602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e925d06e-ffd1-49f2-b016-c2896305a631 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:13.002620 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e925d06e-ffd1-49f2-b016-c2896305a631 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:13.003578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e925d06e-ffd1-49f2-b016-c2896305a631 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:13.004096 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e925d06e-ffd1-49f2-b016-c2896305a631 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:13.007378 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23c81b8-c911-4278-9d9c-56cce57d879d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:13.007996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23c81b8-c911-4278-9d9c-56cce57d879d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:13.008296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23c81b8-c911-4278-9d9c-56cce57d879d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:13.008561 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23c81b8-c911-4278-9d9c-56cce57d879d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:13.016641 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e925d06e-ffd1-49f2-b016-c2896305a631 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:13.019173 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23c81b8-c911-4278-9d9c-56cce57d879d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:13.021295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23c81b8-c911-4278-9d9c-56cce57d879d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:13.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-e925d06e-ffd1-49f2-b016-c2896305a631 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:13.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:13.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:13.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:13.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:13.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:13.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:13.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:13.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:13.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:13.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:13.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:13.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:13.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:13.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:13.023173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:13.024090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:13.024090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:13.024090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:13.024090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:13.024090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:13.024090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:13.024090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:13.024090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:13.024090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:13.024090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:13.024090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:13.024090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:13.024090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:13.024090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:13.024090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:13.024090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:13.024949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:13.024949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:13.024949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:13.024949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:13.024949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:13.024949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:13.024949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:13.024949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:13.024949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:13.024949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:13.024949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:13.024949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:13.027160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 282/560] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:13.028254 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e925d06e-ffd1-49f2-b016-c2896305a631 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:13.041184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-b23c81b8-c911-4278-9d9c-56cce57d879d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:13.041184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:13.041184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:13.041184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:13.041184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:13.041184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:13.041184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:13.041184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:13.041184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:13.041184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:13.041184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:13.041184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:13.041184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:13.041184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:13.041184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:13.041184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:13.042355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:13.043250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:13.043250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:13.043250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:13.043250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 279/561] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:13.043484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e4fdca-1d2a-4ac6-a7d4-5d2a74821d7c None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:13.043705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e4fdca-1d2a-4ac6-a7d4-5d2a74821d7c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:13.044175 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e4fdca-1d2a-4ac6-a7d4-5d2a74821d7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:13.061764 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-41e4fdca-1d2a-4ac6-a7d4-5d2a74821d7c None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:13.062932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-41e4fdca-1d2a-4ac6-a7d4-5d2a74821d7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:13.070318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:13.131822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 280/562] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:13.136666 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e925d06e-ffd1-49f2-b016-c2896305a631 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:13.141047 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e4fdca-1d2a-4ac6-a7d4-5d2a74821d7c None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:13.153105 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:13.175796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87034720-b890-4522-900b-e2534cf09f9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:13.176529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87034720-b890-4522-900b-e2534cf09f9e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:13.176813 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87034720-b890-4522-900b-e2534cf09f9e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:13.177089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87034720-b890-4522-900b-e2534cf09f9e None admin] PATH_INFO: `/v3/users/nova` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:13.184256 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87034720-b890-4522-900b-e2534cf09f9e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:13.185709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87034720-b890-4522-900b-e2534cf09f9e None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:13.197661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-87034720-b890-4522-900b-e2534cf09f9e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:05:13.197661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:13.197661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:13.197661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:13.197661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:13.197661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:13.197661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:13.197661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:13.197661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:13.197661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:13.197661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:13.197661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:13.197661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:13.197661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:13.197661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:13.197661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:13.199764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:13.199764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:13.199764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:13.199764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:13.199764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:13.199764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:13.199764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:13.199764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:13.199764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:05:13.199764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:13.199764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 283/563] 10.222.0.47 () {56 vars in 1093 bytes} [Sun Aug 21 01:05:13 2022] GET /identity/v3/users/nova => generated 82 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:13.200931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4198ea4-5499-4b79-aefe-06478e3904dd None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:13.200931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4198ea4-5499-4b79-aefe-06478e3904dd None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:13.200931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4198ea4-5499-4b79-aefe-06478e3904dd None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:13.206049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 284/564] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:13 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:13.206049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4198ea4-5499-4b79-aefe-06478e3904dd None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:13.257512 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e4fdca-1d2a-4ac6-a7d4-5d2a74821d7c None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:13.304968 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d6cfdf-18e5-46e6-862e-79056be5f3ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:13.305574 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d6cfdf-18e5-46e6-862e-79056be5f3ca None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:13.309077 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d6cfdf-18e5-46e6-862e-79056be5f3ca None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:13.309077 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d6cfdf-18e5-46e6-862e-79056be5f3ca None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:13.309077 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d6cfdf-18e5-46e6-862e-79056be5f3ca None admin] RBAC: Adding query filter params (name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:13.309077 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d6cfdf-18e5-46e6-862e-79056be5f3ca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:13.309530 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d6cfdf-18e5-46e6-862e-79056be5f3ca None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:13.327093 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 281/565] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:13.327093 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d6cfdf-18e5-46e6-862e-79056be5f3ca None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:13.328370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:13.337403 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4198ea4-5499-4b79-aefe-06478e3904dd None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:13.378480 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4314e45-76f1-4f09-9c51-da42c04b1e04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:13.379409 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4314e45-76f1-4f09-9c51-da42c04b1e04 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:13.380293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4314e45-76f1-4f09-9c51-da42c04b1e04 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:13.380628 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4314e45-76f1-4f09-9c51-da42c04b1e04 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:13.381268 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4314e45-76f1-4f09-9c51-da42c04b1e04 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:13.381567 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4314e45-76f1-4f09-9c51-da42c04b1e04 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:13.383051 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4314e45-76f1-4f09-9c51-da42c04b1e04 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:13.399104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 285/566] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:13.400672 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9dabdf-366f-4309-b2b0-c74ff2b45e3e None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:13.400956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9dabdf-366f-4309-b2b0-c74ff2b45e3e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:13.401233 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9dabdf-366f-4309-b2b0-c74ff2b45e3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:13.427297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-1e9dabdf-366f-4309-b2b0-c74ff2b45e3e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:13.428587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1e9dabdf-366f-4309-b2b0-c74ff2b45e3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:13.433183 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:13.466735 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d6cfdf-18e5-46e6-862e-79056be5f3ca None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:13.513175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 286/567] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:13.514416 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9dabdf-366f-4309-b2b0-c74ff2b45e3e None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:13.519761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3b1ff3-cd85-43f4-9bb6-4ee4cd9f1b92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:13.520823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3b1ff3-cd85-43f4-9bb6-4ee4cd9f1b92 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:13.521243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3b1ff3-cd85-43f4-9bb6-4ee4cd9f1b92 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:13.521654 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3b1ff3-cd85-43f4-9bb6-4ee4cd9f1b92 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:13.522899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3b1ff3-cd85-43f4-9bb6-4ee4cd9f1b92 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:13.523376 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3b1ff3-cd85-43f4-9bb6-4ee4cd9f1b92 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:13.525706 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3b1ff3-cd85-43f4-9bb6-4ee4cd9f1b92 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:13.562151 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 282/568] 10.222.0.47 () {56 vars in 1133 bytes} [Sun Aug 21 01:05:13 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:13.562151 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3b1ff3-cd85-43f4-9bb6-4ee4cd9f1b92 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:13.631266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:13.650866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9dabdf-366f-4309-b2b0-c74ff2b45e3e None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:13.686440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3b1ff3-cd85-43f4-9bb6-4ee4cd9f1b92 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:13.695305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e72fab0-691c-430a-8422-341547f5e9c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:13.696322 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e72fab0-691c-430a-8422-341547f5e9c7 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:13.696804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e72fab0-691c-430a-8422-341547f5e9c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:13.697267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e72fab0-691c-430a-8422-341547f5e9c7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:13.713317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e72fab0-691c-430a-8422-341547f5e9c7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:13.714893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e72fab0-691c-430a-8422-341547f5e9c7 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:13.726414 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:13.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-2e72fab0-691c-430a-8422-341547f5e9c7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:13.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:13.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:13.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:13.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:13.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:13.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:13.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:13.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:13.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:13.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:13.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:13.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:13.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:13.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:13.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:13.738205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:13.739916 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:13.739916 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:13.739916 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:13.739916 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 287/569] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:13.739916 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e72fab0-691c-430a-8422-341547f5e9c7 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:13.740286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bb3e34-fe38-4e0e-891a-c0bcf4261495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:13.740286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bb3e34-fe38-4e0e-891a-c0bcf4261495 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:13.740286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bb3e34-fe38-4e0e-891a-c0bcf4261495 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:13.740286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bb3e34-fe38-4e0e-891a-c0bcf4261495 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:13.750744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67bb3e34-fe38-4e0e-891a-c0bcf4261495 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:13.752301 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67bb3e34-fe38-4e0e-891a-c0bcf4261495 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:13.764377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-67bb3e34-fe38-4e0e-891a-c0bcf4261495 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:13.764377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:13.764377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:13.764377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:13.764377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:13.764377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:13.764377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:13.764377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:13.764377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:13.764377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:13.764377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:13.764377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:13.764377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:13.764377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:13.764377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:13.765587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:13.766670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:13.766670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:13.766670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:13.766670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:13.766670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:13.766670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:13.766670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:13.766670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:13.767251 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 283/570] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:13.768058 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bb3e34-fe38-4e0e-891a-c0bcf4261495 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:13.855776 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e72fab0-691c-430a-8422-341547f5e9c7 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:13.871630 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bb3e34-fe38-4e0e-891a-c0bcf4261495 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:13.890985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85a3828-c890-4a28-bf07-2fc84929c358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:13.891726 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85a3828-c890-4a28-bf07-2fc84929c358 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:13.892001 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85a3828-c890-4a28-bf07-2fc84929c358 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:13.892301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85a3828-c890-4a28-bf07-2fc84929c358 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:13.900041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:13.904213 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85a3828-c890-4a28-bf07-2fc84929c358 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:13.906755 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde8ace9-1836-439f-bca5-d5d37ad5a9fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:13.907393 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde8ace9-1836-439f-bca5-d5d37ad5a9fb None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:13.907669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde8ace9-1836-439f-bca5-d5d37ad5a9fb None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:13.907937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde8ace9-1836-439f-bca5-d5d37ad5a9fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:13.908931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-a85a3828-c890-4a28-bf07-2fc84929c358 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:13.908931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:13.908931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:13.908931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:13.908931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:13.908931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:13.908931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:13.908931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:13.908931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:13.908931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:13.908931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:13.908931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:13.908931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:13.908931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:13.908931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:13.908931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:13.910025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:13.910025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:13.910025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:13.910025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:13.910025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:13.910025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:13.910025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:13.910025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:13.910025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:13.910025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:13.910025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:13.910025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:13.910025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:13.910025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:13.910025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:13.910025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:13.911571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:13.911571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:13.911571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:13.911571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:13.911571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:13.911571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:13.911571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:13.911571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:13.911571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:13.911571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:13.911571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:13.911571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:13.911571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 288/571] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:13.911571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85a3828-c890-4a28-bf07-2fc84929c358 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:13.922955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde8ace9-1836-439f-bca5-d5d37ad5a9fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:13.924488 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde8ace9-1836-439f-bca5-d5d37ad5a9fb None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:13.939293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-fde8ace9-1836-439f-bca5-d5d37ad5a9fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:13.939293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:13.939293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:13.939293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:13.939293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:13.939293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:13.939293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:13.939293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:13.939293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:13.939293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:13.939293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:13.939293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:13.939293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:13.939293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:13.939293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:13.939293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:13.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:13.941435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:13.941435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:13.941435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:13.941435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 284/572] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:13.941435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde8ace9-1836-439f-bca5-d5d37ad5a9fb None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:13.988117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:14.021485 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85a3828-c890-4a28-bf07-2fc84929c358 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:14.046280 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde8ace9-1836-439f-bca5-d5d37ad5a9fb None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:14.060828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3b5b29-3807-4b57-a6dc-22386474d79a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:14.061441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3b5b29-3807-4b57-a6dc-22386474d79a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:14.061719 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3b5b29-3807-4b57-a6dc-22386474d79a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:14.061991 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3b5b29-3807-4b57-a6dc-22386474d79a None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:14.062638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3b5b29-3807-4b57-a6dc-22386474d79a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:14.062934 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3b5b29-3807-4b57-a6dc-22386474d79a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:14.064444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3b5b29-3807-4b57-a6dc-22386474d79a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:14.077116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 289/573] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:14.077946 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3b5b29-3807-4b57-a6dc-22386474d79a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:14.080285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063b06b0-a394-4d88-9c32-b5d35078d322 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:14.080899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063b06b0-a394-4d88-9c32-b5d35078d322 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:14.081172 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063b06b0-a394-4d88-9c32-b5d35078d322 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:14.081439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063b06b0-a394-4d88-9c32-b5d35078d322 None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:14.082135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063b06b0-a394-4d88-9c32-b5d35078d322 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:14.082432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063b06b0-a394-4d88-9c32-b5d35078d322 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:14.084501 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063b06b0-a394-4d88-9c32-b5d35078d322 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:14.097853 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 285/574] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:14.099525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063b06b0-a394-4d88-9c32-b5d35078d322 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:14.143602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:14.186199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3b5b29-3807-4b57-a6dc-22386474d79a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:14.203017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063b06b0-a394-4d88-9c32-b5d35078d322 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:14.204869 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:14.224675 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3b39eb-941d-46bd-984c-4d2bfaf22de3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:14.225380 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3b39eb-941d-46bd-984c-4d2bfaf22de3 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:14.225657 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3b39eb-941d-46bd-984c-4d2bfaf22de3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:14.225948 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3b39eb-941d-46bd-984c-4d2bfaf22de3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:14.237748 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3b39eb-941d-46bd-984c-4d2bfaf22de3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:14.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-3c3b39eb-941d-46bd-984c-4d2bfaf22de3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:14.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:14.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:14.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:14.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:14.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:14.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:14.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:14.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:14.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:14.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:14.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:14.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:14.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:14.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:14.242773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:14.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:14.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:14.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:14.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:14.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:14.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:14.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:14.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:14.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:14.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:14.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:14.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:14.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:14.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:14.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:14.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:14.244749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:14.244749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:14.244749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:14.244749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:14.244749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:14.244749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:14.244749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:14.244749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:14.244749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:14.244749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:14.244749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:14.244749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:14.245388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa39085-411e-45d6-9ebd-6c07bb21de0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:14.245478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 290/575] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:14.245561 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa39085-411e-45d6-9ebd-6c07bb21de0f None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:14.245637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3b39eb-941d-46bd-984c-4d2bfaf22de3 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:14.245707 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa39085-411e-45d6-9ebd-6c07bb21de0f None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:14.245707 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa39085-411e-45d6-9ebd-6c07bb21de0f None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:14.246110 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa39085-411e-45d6-9ebd-6c07bb21de0f None admin] RBAC: Adding query filter params (name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:14.246578 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa39085-411e-45d6-9ebd-6c07bb21de0f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:14.248847 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa39085-411e-45d6-9ebd-6c07bb21de0f None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:14.262160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 286/576] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:14.263042 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa39085-411e-45d6-9ebd-6c07bb21de0f None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:14.358466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3b39eb-941d-46bd-984c-4d2bfaf22de3 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:14.371977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa39085-411e-45d6-9ebd-6c07bb21de0f None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:14.375941 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:14.393723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9aad6db-50c0-4b12-8ddb-375673452f2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:14.394329 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9aad6db-50c0-4b12-8ddb-375673452f2a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:14.394599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9aad6db-50c0-4b12-8ddb-375673452f2a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:14.394884 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9aad6db-50c0-4b12-8ddb-375673452f2a None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:14.396249 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9aad6db-50c0-4b12-8ddb-375673452f2a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:14.396546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9aad6db-50c0-4b12-8ddb-375673452f2a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:14.397931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9aad6db-50c0-4b12-8ddb-375673452f2a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:14.410528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 291/577] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:14.411485 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9aad6db-50c0-4b12-8ddb-375673452f2a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:14.425458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ae949e-de1c-476a-9f26-ef88b192a3c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:14.426112 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ae949e-de1c-476a-9f26-ef88b192a3c1 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:14.426411 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ae949e-de1c-476a-9f26-ef88b192a3c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:14.426683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ae949e-de1c-476a-9f26-ef88b192a3c1 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:14.435220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ae949e-de1c-476a-9f26-ef88b192a3c1 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:14.436717 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ae949e-de1c-476a-9f26-ef88b192a3c1 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:14.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-89ae949e-de1c-476a-9f26-ef88b192a3c1 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 21 01:05:14.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:14.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:14.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:14.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:14.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:14.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:14.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:14.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:14.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:14.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:14.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:14.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:14.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:14.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:14.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:14.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:14.451719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:14.451719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:14.451719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:14.451719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:14.451719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:14.451719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:14.451719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:14.451719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:14.451719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 21 01:05:14.451719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:14.451719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 287/578] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:05:14 2022] GET /identity/v3/users/cinder => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:14.451719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ae949e-de1c-476a-9f26-ef88b192a3c1 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:14.478969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:14.529598 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9aad6db-50c0-4b12-8ddb-375673452f2a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:14.559102 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ae949e-de1c-476a-9f26-ef88b192a3c1 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:14.564633 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c7b2f7-1088-45b3-b910-aac1c07ebf24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:14.565258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c7b2f7-1088-45b3-b910-aac1c07ebf24 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:14.565707 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c7b2f7-1088-45b3-b910-aac1c07ebf24 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:14.566051 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c7b2f7-1088-45b3-b910-aac1c07ebf24 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:14.566778 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c7b2f7-1088-45b3-b910-aac1c07ebf24 None admin] RBAC: Adding query filter params (role.id=c2d8bf7283e14857879d03d1f39c0165, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=06a5972a55824711b06670b925322739) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:14.567637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c7b2f7-1088-45b3-b910-aac1c07ebf24 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:14.569093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c7b2f7-1088-45b3-b910-aac1c07ebf24 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:14.574755 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 292/579] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:14 2022] GET /identity/v3/role_assignments?user.id=06a5972a55824711b06670b925322739&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=c2d8bf7283e14857879d03d1f39c0165 => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:14.575545 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c7b2f7-1088-45b3-b910-aac1c07ebf24 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:14.599172 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b0667f-fc5d-4dca-89c9-f9766bf85dec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:14.599809 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b0667f-fc5d-4dca-89c9-f9766bf85dec None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:14.600082 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b0667f-fc5d-4dca-89c9-f9766bf85dec None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:14.600379 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b0667f-fc5d-4dca-89c9-f9766bf85dec None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:14.601718 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b0667f-fc5d-4dca-89c9-f9766bf85dec None admin] RBAC: Adding query filter params (name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:14.602020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b0667f-fc5d-4dca-89c9-f9766bf85dec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:14.603924 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b0667f-fc5d-4dca-89c9-f9766bf85dec None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:14.621969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 288/580] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:14.621969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b0667f-fc5d-4dca-89c9-f9766bf85dec None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:14.626439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:14.697732 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c7b2f7-1088-45b3-b910-aac1c07ebf24 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:14.724909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:14.740711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e246ff5-41c7-4d1e-9874-96ebe34464d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:14.741266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e246ff5-41c7-4d1e-9874-96ebe34464d4 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:14.741544 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e246ff5-41c7-4d1e-9874-96ebe34464d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:14.742630 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e246ff5-41c7-4d1e-9874-96ebe34464d4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:14.743686 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b0667f-fc5d-4dca-89c9-f9766bf85dec None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:14.760388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e246ff5-41c7-4d1e-9874-96ebe34464d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:14.761857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e246ff5-41c7-4d1e-9874-96ebe34464d4 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:14.777583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-5e246ff5-41c7-4d1e-9874-96ebe34464d4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:14.777583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:14.777583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:14.777583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:14.777583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:14.777583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:14.777583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:14.777583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:14.777583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:14.777583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:14.777583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:14.777583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:14.777583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:14.777583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:14.777583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:14.777583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:14.778606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:14.779574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:14.779574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:14.779574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:14.779574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 293/581] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:14.779574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e246ff5-41c7-4d1e-9874-96ebe34464d4 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:14.792587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504f8b1e-9b97-4074-8bd0-56ae9a1a6903 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:14.793728 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504f8b1e-9b97-4074-8bd0-56ae9a1a6903 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:14.794122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504f8b1e-9b97-4074-8bd0-56ae9a1a6903 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:14.794786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504f8b1e-9b97-4074-8bd0-56ae9a1a6903 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:14.802603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504f8b1e-9b97-4074-8bd0-56ae9a1a6903 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:14.804253 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504f8b1e-9b97-4074-8bd0-56ae9a1a6903 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:14.816278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-504f8b1e-9b97-4074-8bd0-56ae9a1a6903 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:14.816278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:14.816278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:14.816278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:14.816278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:14.816278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:14.816278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:14.816278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:14.816278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:14.816278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:14.816278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:14.816278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:14.816278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:14.816278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:14.816278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:14.817780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:14.818913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:14.818913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:14.818913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:14.818913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:14.818913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:14.818913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:14.818913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:14.818913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:14.820045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 289/582] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:14.821060 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504f8b1e-9b97-4074-8bd0-56ae9a1a6903 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:14.898851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:14.907317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e246ff5-41c7-4d1e-9874-96ebe34464d4 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:14.939592 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504f8b1e-9b97-4074-8bd0-56ae9a1a6903 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:14.950703 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27d9375-3254-48f6-8f88-b44463b691f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:14.951337 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27d9375-3254-48f6-8f88-b44463b691f9 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:14.951586 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27d9375-3254-48f6-8f88-b44463b691f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:14.951854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27d9375-3254-48f6-8f88-b44463b691f9 None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:14.952722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27d9375-3254-48f6-8f88-b44463b691f9 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:14.952903 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27d9375-3254-48f6-8f88-b44463b691f9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:14.954338 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27d9375-3254-48f6-8f88-b44463b691f9 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:14.977141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 294/583] 10.222.0.47 () {56 vars in 1137 bytes} [Sun Aug 21 01:05:14 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:14.977995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27d9375-3254-48f6-8f88-b44463b691f9 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:14.988908 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30b1337-8c22-4146-aac7-6f79a02aa441 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:14.989639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30b1337-8c22-4146-aac7-6f79a02aa441 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:14.989916 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30b1337-8c22-4146-aac7-6f79a02aa441 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:14.990186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30b1337-8c22-4146-aac7-6f79a02aa441 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:15.002504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30b1337-8c22-4146-aac7-6f79a02aa441 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:15.003913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30b1337-8c22-4146-aac7-6f79a02aa441 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:15.012507 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:15.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-b30b1337-8c22-4146-aac7-6f79a02aa441 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:15.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:15.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:15.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:15.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:15.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:15.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:15.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:15.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:15.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:15.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:15.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:15.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:15.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:15.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:15.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:15.027756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:15.028920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:15.028920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:15.028920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:15.028920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 290/584] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:15.028920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30b1337-8c22-4146-aac7-6f79a02aa441 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:15.108578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27d9375-3254-48f6-8f88-b44463b691f9 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:15.144581 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30b1337-8c22-4146-aac7-6f79a02aa441 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:15.150253 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18e824c-8980-4ef8-a56e-c004d680956c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:15.150942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e824c-8980-4ef8-a56e-c004d680956c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:15.151312 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e824c-8980-4ef8-a56e-c004d680956c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:15.151654 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e824c-8980-4ef8-a56e-c004d680956c None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:15.152419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18e824c-8980-4ef8-a56e-c004d680956c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:15.152820 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18e824c-8980-4ef8-a56e-c004d680956c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:15.154269 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18e824c-8980-4ef8-a56e-c004d680956c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:15.170938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 295/585] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:15.171942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18e824c-8980-4ef8-a56e-c004d680956c None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:15.184117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e155384-1292-464c-979b-974fdbbd4455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:15.184741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e155384-1292-464c-979b-974fdbbd4455 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:15.185007 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e155384-1292-464c-979b-974fdbbd4455 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:15.185285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e155384-1292-464c-979b-974fdbbd4455 None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:15.185966 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e155384-1292-464c-979b-974fdbbd4455 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:15.186263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e155384-1292-464c-979b-974fdbbd4455 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:15.187679 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e155384-1292-464c-979b-974fdbbd4455 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:15.188638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:15.206009 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 291/586] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:15.206009 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e155384-1292-464c-979b-974fdbbd4455 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:15.293101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:15.326006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18e824c-8980-4ef8-a56e-c004d680956c None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:15.357935 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e155384-1292-464c-979b-974fdbbd4455 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:15.368211 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a18097-18a7-4514-afb7-3065acd01a9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:15.369277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a18097-18a7-4514-afb7-3065acd01a9e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:15.369485 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a18097-18a7-4514-afb7-3065acd01a9e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:15.369888 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a18097-18a7-4514-afb7-3065acd01a9e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:15.382144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a18097-18a7-4514-afb7-3065acd01a9e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:15.383609 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a18097-18a7-4514-afb7-3065acd01a9e None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:15.395353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dc0269-7bc7-4189-85d7-86eb46ae0f06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:15.395948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dc0269-7bc7-4189-85d7-86eb46ae0f06 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:15.396251 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dc0269-7bc7-4189-85d7-86eb46ae0f06 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:15.396518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dc0269-7bc7-4189-85d7-86eb46ae0f06 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:15.397139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dc0269-7bc7-4189-85d7-86eb46ae0f06 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:15.397432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dc0269-7bc7-4189-85d7-86eb46ae0f06 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:15.398041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-32a18097-18a7-4514-afb7-3065acd01a9e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:15.398041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:15.398041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:15.398041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:15.398041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:15.398041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:15.398041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:15.398041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:15.398041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:15.398041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:15.398041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:15.398041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:15.398041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:15.398041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:15.398041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:15.398041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:15.398917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:15.399804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:15.399804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:15.399804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:15.399974 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dc0269-7bc7-4189-85d7-86eb46ae0f06 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:15.400045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 296/587] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:15.400619 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a18097-18a7-4514-afb7-3065acd01a9e None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:15.416799 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 292/588] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:15.417958 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dc0269-7bc7-4189-85d7-86eb46ae0f06 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:15.485433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:15.525354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dc0269-7bc7-4189-85d7-86eb46ae0f06 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:15.527722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a18097-18a7-4514-afb7-3065acd01a9e None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:15.544554 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:15.562616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d704cdd-e4e3-4690-aa39-f3c7fe421bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:15.563276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d704cdd-e4e3-4690-aa39-f3c7fe421bc0 None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:15.563547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d704cdd-e4e3-4690-aa39-f3c7fe421bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:15.564566 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d704cdd-e4e3-4690-aa39-f3c7fe421bc0 None admin] PATH_INFO: `/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/fabdcdc25d1d4ef6bc66bd6cadecaaf8/roles/c2d8bf7283e14857879d03d1f39c0165` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:15.564692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735ea417-4edd-489b-8ce6-35c7c5f7a565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:15.565140 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735ea417-4edd-489b-8ce6-35c7c5f7a565 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:15.565422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735ea417-4edd-489b-8ce6-35c7c5f7a565 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:15.565693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735ea417-4edd-489b-8ce6-35c7c5f7a565 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:15.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735ea417-4edd-489b-8ce6-35c7c5f7a565 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:15.577006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735ea417-4edd-489b-8ce6-35c7c5f7a565 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:15.580749 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d704cdd-e4e3-4690-aa39-f3c7fe421bc0 None admin] RBAC: Authorizing `identity:create_grant(project_id=75a3bf3f3a4244d18fde47d492657db4, user_id=fabdcdc25d1d4ef6bc66bd6cadecaaf8, role_id=c2d8bf7283e14857879d03d1f39c0165)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:15.582278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d704cdd-e4e3-4690-aa39-f3c7fe421bc0 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:15.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-735ea417-4edd-489b-8ce6-35c7c5f7a565 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 21 01:05:15.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:15.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:15.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:15.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:15.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:15.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:15.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:15.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:15.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:15.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:15.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:15.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:15.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:15.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:15.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:15.589945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:15.590954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:15.590954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:15.590954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:15.590954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:15.590954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:15.590954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:15.590954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:15.590954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:15.590954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 21 01:05:15.590954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:15.590954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 297/589] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:15 2022] GET /identity/v3/users/neutron => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:15.590954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735ea417-4edd-489b-8ce6-35c7c5f7a565 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:15.664130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 293/590] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:05:15 2022] PUT /identity/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/fabdcdc25d1d4ef6bc66bd6cadecaaf8/roles/c2d8bf7283e14857879d03d1f39c0165 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:05:15.664130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d704cdd-e4e3-4690-aa39-f3c7fe421bc0 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:15.726904 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735ea417-4edd-489b-8ce6-35c7c5f7a565 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:15.760204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:15.775169 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75104f75-899e-4f75-b72c-cad6262722b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:15.775797 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75104f75-899e-4f75-b72c-cad6262722b4 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:15.776075 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75104f75-899e-4f75-b72c-cad6262722b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:15.776376 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75104f75-899e-4f75-b72c-cad6262722b4 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:15.777661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75104f75-899e-4f75-b72c-cad6262722b4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:15.778029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75104f75-899e-4f75-b72c-cad6262722b4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:15.779753 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75104f75-899e-4f75-b72c-cad6262722b4 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:15.799179 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 298/591] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:15.801262 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75104f75-899e-4f75-b72c-cad6262722b4 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:15.808661 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d704cdd-e4e3-4690-aa39-f3c7fe421bc0 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:15.856926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9f315f-4546-49ad-9fe6-9bf29130ab58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:15.857745 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9f315f-4546-49ad-9fe6-9bf29130ab58 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:15.858030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9f315f-4546-49ad-9fe6-9bf29130ab58 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:15.858431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9f315f-4546-49ad-9fe6-9bf29130ab58 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:15.870355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b9f315f-4546-49ad-9fe6-9bf29130ab58 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:15.872282 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b9f315f-4546-49ad-9fe6-9bf29130ab58 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:15.884607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-0b9f315f-4546-49ad-9fe6-9bf29130ab58 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 21 01:05:15.884607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:15.884607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:15.884607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:15.884607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:15.884607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:15.884607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:15.884607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:15.884607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:15.884607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:15.884607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:15.884607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:15.884607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:15.884607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:15.884607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:15.884607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:15.885791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:15.886945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:15.886945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:15.886945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:15.886945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:15.886945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:15.886945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:15.886945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:15.886945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:15.886945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 21 01:05:15.886945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:15.886945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 294/592] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:05:15 2022] GET /identity/v3/users/glance => generated 84 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:15.886945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9f315f-4546-49ad-9fe6-9bf29130ab58 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:15.920453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:15.944889 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75104f75-899e-4f75-b72c-cad6262722b4 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:15.997657 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a5fd74-129a-4544-bf22-c37794f709aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:15.998559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a5fd74-129a-4544-bf22-c37794f709aa None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:15.998835 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a5fd74-129a-4544-bf22-c37794f709aa None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:15.999109 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a5fd74-129a-4544-bf22-c37794f709aa None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:16.000001 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a5fd74-129a-4544-bf22-c37794f709aa None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:16.000329 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a5fd74-129a-4544-bf22-c37794f709aa None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:16.002315 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a5fd74-129a-4544-bf22-c37794f709aa None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:16.006789 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9f315f-4546-49ad-9fe6-9bf29130ab58 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:16.027201 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 299/593] 10.222.0.47 () {56 vars in 1139 bytes} [Sun Aug 21 01:05:15 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:16.028153 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a5fd74-129a-4544-bf22-c37794f709aa None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:16.045345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a752abff-a8f5-4c20-b20c-9703aa816427 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:16.046146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a752abff-a8f5-4c20-b20c-9703aa816427 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:16.046528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a752abff-a8f5-4c20-b20c-9703aa816427 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:16.046930 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a752abff-a8f5-4c20-b20c-9703aa816427 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:16.055020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a752abff-a8f5-4c20-b20c-9703aa816427 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:16.056525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a752abff-a8f5-4c20-b20c-9703aa816427 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:16.070012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-a752abff-a8f5-4c20-b20c-9703aa816427 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:16.070012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:16.070012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:16.070012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:16.070012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:16.070012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:16.070012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:16.070012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:16.070012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:16.070012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:16.070012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:16.070012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:16.070012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:16.070012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:16.070012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:16.071140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:16.072226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:16.072226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:16.072226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:16.072226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:16.072226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:16.072226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:16.072226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:16.072226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:16.072663 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 295/594] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:16.077398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a752abff-a8f5-4c20-b20c-9703aa816427 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:16.083133 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:16.160973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a5fd74-129a-4544-bf22-c37794f709aa None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:16.173091 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:16.188745 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a752abff-a8f5-4c20-b20c-9703aa816427 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:16.209777 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c177d1f-f1de-4ba4-9cb4-08d2859e30a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:16.210395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c177d1f-f1de-4ba4-9cb4-08d2859e30a5 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:16.210674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c177d1f-f1de-4ba4-9cb4-08d2859e30a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:16.210944 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c177d1f-f1de-4ba4-9cb4-08d2859e30a5 None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:16.211680 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c177d1f-f1de-4ba4-9cb4-08d2859e30a5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:16.211978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c177d1f-f1de-4ba4-9cb4-08d2859e30a5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:16.214013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c177d1f-f1de-4ba4-9cb4-08d2859e30a5 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:16.231788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7693b0cd-1b9f-4ba8-b724-1eb755b915f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:16.232697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7693b0cd-1b9f-4ba8-b724-1eb755b915f0 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:16.233078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7693b0cd-1b9f-4ba8-b724-1eb755b915f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:16.233524 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7693b0cd-1b9f-4ba8-b724-1eb755b915f0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:16.236980 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 300/595] 10.222.0.47 () {56 vars in 1137 bytes} [Sun Aug 21 01:05:16 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:16.237856 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c177d1f-f1de-4ba4-9cb4-08d2859e30a5 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:16.245819 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7693b0cd-1b9f-4ba8-b724-1eb755b915f0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:16.247486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7693b0cd-1b9f-4ba8-b724-1eb755b915f0 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:16.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-7693b0cd-1b9f-4ba8-b724-1eb755b915f0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:16.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:16.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:16.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:16.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:16.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:16.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:16.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:16.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:16.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:16.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:16.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:16.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:16.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:16.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:16.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:16.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:16.267673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:16.267673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:16.267673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:16.267673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 296/596] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:16.272463 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7693b0cd-1b9f-4ba8-b724-1eb755b915f0 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:16.391532 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7693b0cd-1b9f-4ba8-b724-1eb755b915f0 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:16.394683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:16.399053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c177d1f-f1de-4ba4-9cb4-08d2859e30a5 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:16.429183 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ff84ef-968b-47ad-ba1f-14eeb785a5ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:16.429906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ff84ef-968b-47ad-ba1f-14eeb785a5ca None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:16.430149 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ff84ef-968b-47ad-ba1f-14eeb785a5ca None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:16.430420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ff84ef-968b-47ad-ba1f-14eeb785a5ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:16.440345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:16.443713 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ff84ef-968b-47ad-ba1f-14eeb785a5ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:16.445481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ff84ef-968b-47ad-ba1f-14eeb785a5ca None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:16.447813 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee80711f-48b4-45f6-98c6-015d05f36404 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:16.458155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee80711f-48b4-45f6-98c6-015d05f36404 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:16.458155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee80711f-48b4-45f6-98c6-015d05f36404 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:16.458155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee80711f-48b4-45f6-98c6-015d05f36404 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:16.458155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee80711f-48b4-45f6-98c6-015d05f36404 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:16.458155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee80711f-48b4-45f6-98c6-015d05f36404 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:16.458155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee80711f-48b4-45f6-98c6-015d05f36404 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:16.462198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-11ff84ef-968b-47ad-ba1f-14eeb785a5ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:16.462198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:16.462198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:16.462198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:16.462198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:16.462198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:16.462198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:16.462198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:16.462198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:16.462198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:16.462198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:16.462198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:16.462198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:16.462198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:16.462198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:16.462198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:16.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:16.464598 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:16.464598 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:16.464598 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:16.465863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 297/597] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:16.466766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ff84ef-968b-47ad-ba1f-14eeb785a5ca None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:16.471794 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 301/598] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:16.472898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee80711f-48b4-45f6-98c6-015d05f36404 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:16.594762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ff84ef-968b-47ad-ba1f-14eeb785a5ca None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:16.603169 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee80711f-48b4-45f6-98c6-015d05f36404 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:16.632762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12da0bc-0509-40ac-98ce-b91f71513121 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:16.633468 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12da0bc-0509-40ac-98ce-b91f71513121 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:16.633835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12da0bc-0509-40ac-98ce-b91f71513121 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:16.634119 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12da0bc-0509-40ac-98ce-b91f71513121 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:16.634756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12da0bc-0509-40ac-98ce-b91f71513121 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:16.635055 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12da0bc-0509-40ac-98ce-b91f71513121 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:16.636437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12da0bc-0509-40ac-98ce-b91f71513121 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:16.646002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24ee1ae-a3af-41c8-83df-d1c010139fbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:16.647044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24ee1ae-a3af-41c8-83df-d1c010139fbd None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:16.647323 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24ee1ae-a3af-41c8-83df-d1c010139fbd None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:16.647827 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24ee1ae-a3af-41c8-83df-d1c010139fbd None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:16.648412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24ee1ae-a3af-41c8-83df-d1c010139fbd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:16.648713 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24ee1ae-a3af-41c8-83df-d1c010139fbd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:16.649780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 298/599] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:16.650070 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24ee1ae-a3af-41c8-83df-d1c010139fbd None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:16.654787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12da0bc-0509-40ac-98ce-b91f71513121 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:16.668024 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 302/600] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:16.668024 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aebc13-4871-4726-89f5-8f127df8ade9 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:16.668024 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aebc13-4871-4726-89f5-8f127df8ade9 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:16.668024 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aebc13-4871-4726-89f5-8f127df8ade9 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:16.671713 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 303/601] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:16 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:16.672808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aebc13-4871-4726-89f5-8f127df8ade9 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:16.697023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:16.717615 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:16.773005 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12da0bc-0509-40ac-98ce-b91f71513121 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:16.791184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aebc13-4871-4726-89f5-8f127df8ade9 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:16.813647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f691dd75-bcc8-44e9-b36c-030e167ae89b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:16.813647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691dd75-bcc8-44e9-b36c-030e167ae89b None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:16.814078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691dd75-bcc8-44e9-b36c-030e167ae89b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:16.814419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691dd75-bcc8-44e9-b36c-030e167ae89b None admin] PATH_INFO: `/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/b953036b8b774f25bda59cec96b21c27/roles/c2d8bf7283e14857879d03d1f39c0165` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:16.836115 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f691dd75-bcc8-44e9-b36c-030e167ae89b None admin] RBAC: Authorizing `identity:create_grant(project_id=75a3bf3f3a4244d18fde47d492657db4, user_id=b953036b8b774f25bda59cec96b21c27, role_id=c2d8bf7283e14857879d03d1f39c0165)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:16.837908 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f691dd75-bcc8-44e9-b36c-030e167ae89b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:16.839343 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c746d4-3119-4681-8bae-ccb9b3675a54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:16.840299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c746d4-3119-4681-8bae-ccb9b3675a54 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:16.840779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c746d4-3119-4681-8bae-ccb9b3675a54 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:16.841222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c746d4-3119-4681-8bae-ccb9b3675a54 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:16.851094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c746d4-3119-4681-8bae-ccb9b3675a54 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:16.852624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c746d4-3119-4681-8bae-ccb9b3675a54 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:16.865957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-54c746d4-3119-4681-8bae-ccb9b3675a54 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:16.865957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:16.865957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:16.865957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:16.865957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:16.865957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:16.865957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:16.865957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:16.865957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:16.865957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:16.865957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:16.865957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:16.865957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:16.865957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:16.865957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:16.867245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:16.868166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:16.868166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:16.868166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:16.868166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:16.868166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:16.868166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:16.868166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:16.868166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:16.868166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 304/602] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:16.868166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c746d4-3119-4681-8bae-ccb9b3675a54 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:16.912085 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 299/603] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:05:16 2022] PUT /identity/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/b953036b8b774f25bda59cec96b21c27/roles/c2d8bf7283e14857879d03d1f39c0165 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:05:16.914360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070a25cd-8751-448a-b08c-96fa1ffc4b22 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:16.914854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070a25cd-8751-448a-b08c-96fa1ffc4b22 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:16.915366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070a25cd-8751-448a-b08c-96fa1ffc4b22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:16.935908 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-070a25cd-8751-448a-b08c-96fa1ffc4b22 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:16.937567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-070a25cd-8751-448a-b08c-96fa1ffc4b22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:16.993830 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:17.023803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c746d4-3119-4681-8bae-ccb9b3675a54 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:17.037249 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 300/604] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:17.038849 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070a25cd-8751-448a-b08c-96fa1ffc4b22 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:17.083411 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:17.120260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f3ec25-0be1-4168-bead-20faafbd0d67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:17.121085 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f3ec25-0be1-4168-bead-20faafbd0d67 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:17.121527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f3ec25-0be1-4168-bead-20faafbd0d67 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:17.121879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f3ec25-0be1-4168-bead-20faafbd0d67 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:17.132887 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f3ec25-0be1-4168-bead-20faafbd0d67 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:17.134466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f3ec25-0be1-4168-bead-20faafbd0d67 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:17.152520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-01f3ec25-0be1-4168-bead-20faafbd0d67 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:17.152520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:17.152520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:17.152520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:17.152520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:17.152520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:17.152520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:17.152520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:17.152520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:17.152520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:17.152520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:17.152520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:17.152520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:17.152520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:17.152520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:17.153664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:17.154780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:17.154780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:17.154780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:17.154780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:17.154780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:17.154780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:17.154780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:17.154780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:17.157130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 305/605] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:17.162980 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f3ec25-0be1-4168-bead-20faafbd0d67 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:17.184039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070a25cd-8751-448a-b08c-96fa1ffc4b22 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:17.232664 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0998c2a9-c9a9-4a76-9287-ae2128031981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:17.233622 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0998c2a9-c9a9-4a76-9287-ae2128031981 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:17.233971 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0998c2a9-c9a9-4a76-9287-ae2128031981 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:17.234707 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0998c2a9-c9a9-4a76-9287-ae2128031981 None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:17.235485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0998c2a9-c9a9-4a76-9287-ae2128031981 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:17.235860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0998c2a9-c9a9-4a76-9287-ae2128031981 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:17.237387 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0998c2a9-c9a9-4a76-9287-ae2128031981 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:17.256261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 301/606] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:17.264094 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0998c2a9-c9a9-4a76-9287-ae2128031981 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:17.342740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f3ec25-0be1-4168-bead-20faafbd0d67 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:17.403308 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95640197-2efc-4c01-8c6c-dc876f683746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:17.404279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95640197-2efc-4c01-8c6c-dc876f683746 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:17.404562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95640197-2efc-4c01-8c6c-dc876f683746 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:17.404828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95640197-2efc-4c01-8c6c-dc876f683746 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:17.405530 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95640197-2efc-4c01-8c6c-dc876f683746 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:17.405828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95640197-2efc-4c01-8c6c-dc876f683746 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:17.407251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95640197-2efc-4c01-8c6c-dc876f683746 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:17.417636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:17.421241 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 306/607] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:17.422253 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95640197-2efc-4c01-8c6c-dc876f683746 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:17.433737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:17.445973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0998c2a9-c9a9-4a76-9287-ae2128031981 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:17.492485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8056e870-7fae-4302-96aa-4649fe514b71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:17.494073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8056e870-7fae-4302-96aa-4649fe514b71 None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:17.496512 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8056e870-7fae-4302-96aa-4649fe514b71 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:17.497176 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8056e870-7fae-4302-96aa-4649fe514b71 None admin] PATH_INFO: `/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/b532ddab1f5e467989edaa9e4b74bd72/roles/c2d8bf7283e14857879d03d1f39c0165` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:17.517855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8056e870-7fae-4302-96aa-4649fe514b71 None admin] RBAC: Authorizing `identity:create_grant(project_id=75a3bf3f3a4244d18fde47d492657db4, user_id=b532ddab1f5e467989edaa9e4b74bd72, role_id=c2d8bf7283e14857879d03d1f39c0165)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:17.520674 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8056e870-7fae-4302-96aa-4649fe514b71 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:17.535663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95640197-2efc-4c01-8c6c-dc876f683746 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:17.574545 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e5393a-a20f-42e3-87d4-33f708064e44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:17.575327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e5393a-a20f-42e3-87d4-33f708064e44 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:17.575693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e5393a-a20f-42e3-87d4-33f708064e44 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:17.576060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e5393a-a20f-42e3-87d4-33f708064e44 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:17.588029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e5393a-a20f-42e3-87d4-33f708064e44 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:17.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-55e5393a-a20f-42e3-87d4-33f708064e44 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:17.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:17.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:17.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:17.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:17.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:17.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:17.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:17.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:17.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:17.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:17.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:17.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:17.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:17.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:17.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:17.594405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:17.594405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:17.594405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:17.594405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:17.594405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:17.594405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:17.594405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:17.594405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:17.594405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:17.594405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:17.594405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:17.594405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:17.594405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:17.594405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:17.594405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:17.594405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:17.595571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:17.595571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:17.595571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:17.595571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:17.595571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:17.595571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:17.595571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:17.595571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:17.595571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:17.595571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:17.595571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:17.595571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:17.600246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 307/608] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:17.600246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e5393a-a20f-42e3-87d4-33f708064e44 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:17.636646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 302/609] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:05:17 2022] PUT /identity/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/b532ddab1f5e467989edaa9e4b74bd72/roles/c2d8bf7283e14857879d03d1f39c0165 => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:05:17.637688 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8056e870-7fae-4302-96aa-4649fe514b71 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:17.693071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:17.758286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e5393a-a20f-42e3-87d4-33f708064e44 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:17.793697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8056e870-7fae-4302-96aa-4649fe514b71 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:17.803856 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ed3766-be2d-487c-b227-5e4cc1ef1e5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:17.805596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ed3766-be2d-487c-b227-5e4cc1ef1e5f None admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:17.806028 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ed3766-be2d-487c-b227-5e4cc1ef1e5f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:17.806838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ed3766-be2d-487c-b227-5e4cc1ef1e5f None admin] PATH_INFO: `/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/a6365b2328f34ab9b1b73ececbf5cc30/roles/c2d8bf7283e14857879d03d1f39c0165` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:17.830587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ed3766-be2d-487c-b227-5e4cc1ef1e5f None admin] RBAC: Authorizing `identity:create_grant(project_id=75a3bf3f3a4244d18fde47d492657db4, user_id=a6365b2328f34ab9b1b73ececbf5cc30, role_id=c2d8bf7283e14857879d03d1f39c0165)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:17.831868 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ed3766-be2d-487c-b227-5e4cc1ef1e5f None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:17.837180 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5424466b-38c7-4b42-9f1d-43a4ed92e0a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:17.837845 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5424466b-38c7-4b42-9f1d-43a4ed92e0a5 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:17.838845 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5424466b-38c7-4b42-9f1d-43a4ed92e0a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:17.838845 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5424466b-38c7-4b42-9f1d-43a4ed92e0a5 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:17.839131 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5424466b-38c7-4b42-9f1d-43a4ed92e0a5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:17.839433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5424466b-38c7-4b42-9f1d-43a4ed92e0a5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:17.840795 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5424466b-38c7-4b42-9f1d-43a4ed92e0a5 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:17.850157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 303/610] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:17 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:17.851364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826a2aaa-8d18-4670-b197-caabd3f437b5 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:17.851627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826a2aaa-8d18-4670-b197-caabd3f437b5 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:17.851895 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826a2aaa-8d18-4670-b197-caabd3f437b5 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:17.853452 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 304/611] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:17.856299 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826a2aaa-8d18-4670-b197-caabd3f437b5 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:17.860609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:17.903572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 308/612] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:05:17 2022] PUT /identity/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/a6365b2328f34ab9b1b73ececbf5cc30/roles/c2d8bf7283e14857879d03d1f39c0165 => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:05:17.904830 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1b456b-6bed-4aac-a317-c36876918371 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:17.905110 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1b456b-6bed-4aac-a317-c36876918371 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:17.905380 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1b456b-6bed-4aac-a317-c36876918371 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:17.938903 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-6e1b456b-6bed-4aac-a317-c36876918371 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:17.940279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6e1b456b-6bed-4aac-a317-c36876918371 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:18.018836 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826a2aaa-8d18-4670-b197-caabd3f437b5 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:18.059407 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51d1471-7eb3-4d5d-88d6-fc3c97b21395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:18.060200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51d1471-7eb3-4d5d-88d6-fc3c97b21395 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:18.060524 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51d1471-7eb3-4d5d-88d6-fc3c97b21395 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:18.060859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51d1471-7eb3-4d5d-88d6-fc3c97b21395 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:18.074054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51d1471-7eb3-4d5d-88d6-fc3c97b21395 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:18.075733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51d1471-7eb3-4d5d-88d6-fc3c97b21395 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:18.097525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-f51d1471-7eb3-4d5d-88d6-fc3c97b21395 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:18.097525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:18.097525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:18.097525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:18.097525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:18.097525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:18.097525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:18.097525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:18.097525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:18.097525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:18.097525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:18.097525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:18.097525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:18.097525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:18.097525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:18.097525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:18.099139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:18.100030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:18.100030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:18.100030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:18.100030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 305/613] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:18.101014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 309/614] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:18.105074 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51d1471-7eb3-4d5d-88d6-fc3c97b21395 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:18.168502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:18.214364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51d1471-7eb3-4d5d-88d6-fc3c97b21395 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:18.250321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9e667a-017d-4d95-93d7-1fa52fe57190 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:18.250931 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e667a-017d-4d95-93d7-1fa52fe57190 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:18.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e667a-017d-4d95-93d7-1fa52fe57190 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:18.251478 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e667a-017d-4d95-93d7-1fa52fe57190 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:18.252139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9e667a-017d-4d95-93d7-1fa52fe57190 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:18.252447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9e667a-017d-4d95-93d7-1fa52fe57190 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:18.253890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9e667a-017d-4d95-93d7-1fa52fe57190 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:18.268617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 306/615] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:18.274703 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1b456b-6bed-4aac-a317-c36876918371 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:18.358209 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:18.431725 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9e667a-017d-4d95-93d7-1fa52fe57190 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:18.451193 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1b456b-6bed-4aac-a317-c36876918371 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:18.540918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cf99ff-ae5e-46b1-8188-e091a2dff8a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:18.541473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cf99ff-ae5e-46b1-8188-e091a2dff8a5 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:18.541747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cf99ff-ae5e-46b1-8188-e091a2dff8a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:18.542017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cf99ff-ae5e-46b1-8188-e091a2dff8a5 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:18.556128 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cf99ff-ae5e-46b1-8188-e091a2dff8a5 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:18.559959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cf99ff-ae5e-46b1-8188-e091a2dff8a5 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:18.563212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9e667a-017d-4d95-93d7-1fa52fe57190 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:18.583947 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:18.587286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-47cf99ff-ae5e-46b1-8188-e091a2dff8a5 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 21 01:05:18.587286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:18.587286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:18.587286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:18.587286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:18.587286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:18.587286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:18.587286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:18.587286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:18.587286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:18.587286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:18.587286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:18.587286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:18.587286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:18.587286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:18.587286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:18.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:18.596731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:18.596731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:18.596731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:18.596731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:18.596731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:18.596731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:18.596731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:18.596731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:18.596731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 21 01:05:18.596731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:18.596731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 310/616] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:05:18 2022] GET /identity/v3/users/swift => generated 83 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:18.596731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cf99ff-ae5e-46b1-8188-e091a2dff8a5 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:18.640863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1db9ec6-457f-4a4b-b9d4-b1f497faa6ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:18.641522 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1db9ec6-457f-4a4b-b9d4-b1f497faa6ef None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:18.641804 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1db9ec6-457f-4a4b-b9d4-b1f497faa6ef None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:18.642076 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1db9ec6-457f-4a4b-b9d4-b1f497faa6ef None admin] PATH_INFO: `/v3/roles/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:18.692987 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1db9ec6-457f-4a4b-b9d4-b1f497faa6ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:18.697744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-b1db9ec6-457f-4a4b-b9d4-b1f497faa6ef None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:18.697744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:18.697744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:18.697744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:18.697744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:18.697744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:18.697744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:18.697744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:18.697744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:18.697744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:18.697744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:18.697744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:18.697744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:18.697744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:18.697744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:18.697744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:18.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:18.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:18.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:18.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:18.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:18.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:18.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:18.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:18.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:18.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:18.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:18.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:18.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:18.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:18.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:18.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:18.699907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:18.699907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:18.699907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:18.699907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:18.699907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:18.699907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:18.699907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:18.699907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:18.699907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:18.699907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:18.699907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:18.699907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:18.699907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 307/617] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:18 2022] GET /identity/v3/roles/service => generated 85 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:18.709070 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1db9ec6-457f-4a4b-b9d4-b1f497faa6ef None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:18.716731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cf99ff-ae5e-46b1-8188-e091a2dff8a5 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:18.725657 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:18.755333 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fa805e-ec1f-4ebe-b6bc-d0c689421230 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:18.756014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fa805e-ec1f-4ebe-b6bc-d0c689421230 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:18.756257 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fa805e-ec1f-4ebe-b6bc-d0c689421230 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:18.756528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fa805e-ec1f-4ebe-b6bc-d0c689421230 None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:18.757737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73fa805e-ec1f-4ebe-b6bc-d0c689421230 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:18.758044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73fa805e-ec1f-4ebe-b6bc-d0c689421230 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:18.759489 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73fa805e-ec1f-4ebe-b6bc-d0c689421230 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:18.779631 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 311/618] 10.222.0.47 () {56 vars in 1135 bytes} [Sun Aug 21 01:05:18 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:18.784818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fa805e-ec1f-4ebe-b6bc-d0c689421230 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:18.930258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fa805e-ec1f-4ebe-b6bc-d0c689421230 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:18.977396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1db9ec6-457f-4a4b-b9d4-b1f497faa6ef None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:18.997187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09ba6f3-f2ef-4175-9b90-2623dcff25a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:18.997827 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09ba6f3-f2ef-4175-9b90-2623dcff25a6 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:18.998107 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09ba6f3-f2ef-4175-9b90-2623dcff25a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:18.998576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09ba6f3-f2ef-4175-9b90-2623dcff25a6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:19.019237 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09ba6f3-f2ef-4175-9b90-2623dcff25a6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:19.021267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09ba6f3-f2ef-4175-9b90-2623dcff25a6 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:19.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-b09ba6f3-f2ef-4175-9b90-2623dcff25a6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:19.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:19.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:19.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:19.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:19.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:19.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:19.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:19.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:19.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:19.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:19.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:19.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:19.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:19.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:19.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:19.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:19.052187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:19.052187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:19.052187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:19.052187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 312/619] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:19.052414 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049218f7-4de1-451c-b6b7-2e64818d1d55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:19.052957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049218f7-4de1-451c-b6b7-2e64818d1d55 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:19.056127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049218f7-4de1-451c-b6b7-2e64818d1d55 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:19.056127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049218f7-4de1-451c-b6b7-2e64818d1d55 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:19.056127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049218f7-4de1-451c-b6b7-2e64818d1d55 None admin] RBAC: Adding query filter params (name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:19.056127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049218f7-4de1-451c-b6b7-2e64818d1d55 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:19.056127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049218f7-4de1-451c-b6b7-2e64818d1d55 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:19.057118 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09ba6f3-f2ef-4175-9b90-2623dcff25a6 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:19.076465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 308/620] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 368 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:19.080475 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:19.087209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049218f7-4de1-451c-b6b7-2e64818d1d55 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:19.173297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:19.225613 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049218f7-4de1-451c-b6b7-2e64818d1d55 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:19.264188 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed33d71-bb19-4153-92fa-0d4c251075c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:19.264832 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed33d71-bb19-4153-92fa-0d4c251075c8 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:19.265110 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed33d71-bb19-4153-92fa-0d4c251075c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:19.265398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed33d71-bb19-4153-92fa-0d4c251075c8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:19.277301 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed33d71-bb19-4153-92fa-0d4c251075c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:19.278826 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed33d71-bb19-4153-92fa-0d4c251075c8 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:19.285260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09ba6f3-f2ef-4175-9b90-2623dcff25a6 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:19.294497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-8ed33d71-bb19-4153-92fa-0d4c251075c8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:19.294497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:19.294497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:19.294497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:19.294497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:19.294497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:19.294497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:19.294497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:19.294497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:19.294497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:19.294497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:19.294497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:19.294497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:19.294497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:19.294497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:19.294497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:19.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:19.299079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:19.299079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:19.299079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:19.299079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 309/621] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:19.301941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07087fc-17a0-4f09-844b-346a4161e67a None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:19.302222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07087fc-17a0-4f09-844b-346a4161e67a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:19.302492 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07087fc-17a0-4f09-844b-346a4161e67a None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:19.304082 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 310/622] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:19 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:19.304912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07087fc-17a0-4f09-844b-346a4161e67a None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:19.335854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e85a753-dfd3-4feb-b98b-67dffcb9174c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:19.336559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e85a753-dfd3-4feb-b98b-67dffcb9174c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:19.337118 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e85a753-dfd3-4feb-b98b-67dffcb9174c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:19.337460 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e85a753-dfd3-4feb-b98b-67dffcb9174c None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:19.340783 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e85a753-dfd3-4feb-b98b-67dffcb9174c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:19.341100 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e85a753-dfd3-4feb-b98b-67dffcb9174c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:19.345038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e85a753-dfd3-4feb-b98b-67dffcb9174c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:19.365990 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 313/623] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:19.365990 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541a28b1-1485-4711-bc00-c750efd6178e None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:19.365990 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541a28b1-1485-4711-bc00-c750efd6178e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:19.365990 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541a28b1-1485-4711-bc00-c750efd6178e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:19.385801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-541a28b1-1485-4711-bc00-c750efd6178e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:19.387774 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-541a28b1-1485-4711-bc00-c750efd6178e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:19.427203 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:19.440866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07087fc-17a0-4f09-844b-346a4161e67a None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:19.477988 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:19.478813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac51473-df5a-4b98-a998-c0813c43e972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:19.479722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac51473-df5a-4b98-a998-c0813c43e972 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:19.480228 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac51473-df5a-4b98-a998-c0813c43e972 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:19.480753 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac51473-df5a-4b98-a998-c0813c43e972 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:19.481610 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac51473-df5a-4b98-a998-c0813c43e972 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:19.482118 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac51473-df5a-4b98-a998-c0813c43e972 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:19.488284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 314/624] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:19.488284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541a28b1-1485-4711-bc00-c750efd6178e None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:19.489716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac51473-df5a-4b98-a998-c0813c43e972 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:19.511456 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 311/625] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:19.517367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac51473-df5a-4b98-a998-c0813c43e972 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:19.611706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541a28b1-1485-4711-bc00-c750efd6178e None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:19.654506 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d5a4cf-3955-425a-8553-3cc53dde7909 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:19.655529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d5a4cf-3955-425a-8553-3cc53dde7909 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:19.656003 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d5a4cf-3955-425a-8553-3cc53dde7909 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:19.656498 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d5a4cf-3955-425a-8553-3cc53dde7909 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:19.665651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d5a4cf-3955-425a-8553-3cc53dde7909 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:19.672125 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d5a4cf-3955-425a-8553-3cc53dde7909 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:19.681088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-19d5a4cf-3955-425a-8553-3cc53dde7909 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:19.681088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:19.681088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:19.681088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:19.681088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:19.681088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:19.681088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:19.681088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:19.681088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:19.681088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:19.681088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:19.681088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:19.681088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:19.681088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:19.681088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:19.682072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:19.683040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:19.683040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:19.683040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:19.683040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:19.683040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:19.683040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:19.683040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:19.683040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:19.683040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 315/626] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:19.688426 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d5a4cf-3955-425a-8553-3cc53dde7909 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:19.719466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac51473-df5a-4b98-a998-c0813c43e972 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:19.760477 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:19.814133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b214b5c4-658f-4311-94d3-225fda2a9145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:19.815644 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b214b5c4-658f-4311-94d3-225fda2a9145 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:19.816301 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b214b5c4-658f-4311-94d3-225fda2a9145 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:19.817667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b214b5c4-658f-4311-94d3-225fda2a9145 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:19.819228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d5a4cf-3955-425a-8553-3cc53dde7909 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:19.827913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b214b5c4-658f-4311-94d3-225fda2a9145 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:19.830209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b214b5c4-658f-4311-94d3-225fda2a9145 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:19.846584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-b214b5c4-658f-4311-94d3-225fda2a9145 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:05:19.846584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:19.846584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:19.846584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:19.846584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:19.846584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:19.846584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:19.846584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:19.846584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:19.846584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:19.846584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:19.846584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:19.846584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:19.846584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:19.846584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:19.846584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:19.848802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:19.848802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:19.848802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:19.848802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:19.848802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:19.848802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:19.848802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:19.848802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:19.848802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:05:19.848802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:19.852368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 312/627] 10.222.0.47 () {56 vars in 1093 bytes} [Sun Aug 21 01:05:19 2022] GET /identity/v3/users/nova => generated 82 bytes in 335 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:19.854126 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0994fc96-b2b5-4b00-980c-2496166d12b4 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:19.855055 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0994fc96-b2b5-4b00-980c-2496166d12b4 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:19.855861 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0994fc96-b2b5-4b00-980c-2496166d12b4 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:19.857694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 313/628] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:19.858627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0994fc96-b2b5-4b00-980c-2496166d12b4 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:19.864252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2e5694-d7ee-4197-a037-bc5cc4612fb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:19.864877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2e5694-d7ee-4197-a037-bc5cc4612fb3 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:19.865155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2e5694-d7ee-4197-a037-bc5cc4612fb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:19.865427 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2e5694-d7ee-4197-a037-bc5cc4612fb3 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:19.866136 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2e5694-d7ee-4197-a037-bc5cc4612fb3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:19.866439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2e5694-d7ee-4197-a037-bc5cc4612fb3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:19.867876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2e5694-d7ee-4197-a037-bc5cc4612fb3 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:19.876820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:19.881182 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 316/629] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:19.882413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbddc1ac-5564-4d27-bff3-503c23e9ef69 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:19.882698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbddc1ac-5564-4d27-bff3-503c23e9ef69 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:19.882964 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbddc1ac-5564-4d27-bff3-503c23e9ef69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:19.901401 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-bbddc1ac-5564-4d27-bff3-503c23e9ef69 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:19.902773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-bbddc1ac-5564-4d27-bff3-503c23e9ef69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:19.976861 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 317/630] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:19.977819 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbddc1ac-5564-4d27-bff3-503c23e9ef69 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:19.990884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0994fc96-b2b5-4b00-980c-2496166d12b4 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:20.042017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:20.045035 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404aceef-d5c0-4708-8d4d-ee44b899a2a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:20.046408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404aceef-d5c0-4708-8d4d-ee44b899a2a5 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:20.047124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404aceef-d5c0-4708-8d4d-ee44b899a2a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:20.047540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404aceef-d5c0-4708-8d4d-ee44b899a2a5 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:20.048654 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404aceef-d5c0-4708-8d4d-ee44b899a2a5 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:20.049120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404aceef-d5c0-4708-8d4d-ee44b899a2a5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:20.051309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404aceef-d5c0-4708-8d4d-ee44b899a2a5 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:20.075176 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 314/631] 10.222.0.47 () {56 vars in 1133 bytes} [Sun Aug 21 01:05:19 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:20.077192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404aceef-d5c0-4708-8d4d-ee44b899a2a5 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:20.101668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbddc1ac-5564-4d27-bff3-503c23e9ef69 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:20.146112 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd377db3-6050-4a47-b76e-f3f74e04241b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:20.147136 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd377db3-6050-4a47-b76e-f3f74e04241b None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:20.147618 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd377db3-6050-4a47-b76e-f3f74e04241b None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:20.148721 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd377db3-6050-4a47-b76e-f3f74e04241b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:20.164163 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd377db3-6050-4a47-b76e-f3f74e04241b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:20.166037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:20.173046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-cd377db3-6050-4a47-b76e-f3f74e04241b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:20.173046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:20.173046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:20.173046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:20.173046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:20.173046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:20.173046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:20.173046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:20.173046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:20.173046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:20.173046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:20.173046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:20.173046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:20.173046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:20.173046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:20.173046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:20.174038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:20.174038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:20.174038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:20.174038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:20.174038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:20.174038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:20.174038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:20.174038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:20.174038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:20.174038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:20.174038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:20.174038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:20.174038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:20.174038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:20.174038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:20.174038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:20.174893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:20.174893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:20.174893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:20.174893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:20.174893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:20.174893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:20.174893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:20.174893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:20.174893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:20.174893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:20.174893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:20.174893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:20.180262 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 318/632] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:20.181367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd377db3-6050-4a47-b76e-f3f74e04241b None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:20.210666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404aceef-d5c0-4708-8d4d-ee44b899a2a5 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:20.250588 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec22435-a6b8-45cb-b1fc-eedf0394d1bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:20.251363 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec22435-a6b8-45cb-b1fc-eedf0394d1bc None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:20.251640 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec22435-a6b8-45cb-b1fc-eedf0394d1bc None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:20.251920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec22435-a6b8-45cb-b1fc-eedf0394d1bc None admin] PATH_INFO: `/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/06a5972a55824711b06670b925322739/roles/c2d8bf7283e14857879d03d1f39c0165` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:20.268303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec22435-a6b8-45cb-b1fc-eedf0394d1bc None admin] RBAC: Authorizing `identity:create_grant(project_id=75a3bf3f3a4244d18fde47d492657db4, user_id=06a5972a55824711b06670b925322739, role_id=c2d8bf7283e14857879d03d1f39c0165)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:20.270041 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec22435-a6b8-45cb-b1fc-eedf0394d1bc None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:20.308787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd377db3-6050-4a47-b76e-f3f74e04241b None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:20.319832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:20.343534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1304c91e-b6e4-41b5-b11c-81d4f37daa63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:20.344151 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1304c91e-b6e4-41b5-b11c-81d4f37daa63 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:20.344432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1304c91e-b6e4-41b5-b11c-81d4f37daa63 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:20.344707 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1304c91e-b6e4-41b5-b11c-81d4f37daa63 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:20.365902 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 315/633] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:05:20 2022] PUT /identity/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/06a5972a55824711b06670b925322739/roles/c2d8bf7283e14857879d03d1f39c0165 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:05:20.380408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec22435-a6b8-45cb-b1fc-eedf0394d1bc None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:20.421153 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1304c91e-b6e4-41b5-b11c-81d4f37daa63 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:20.422658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1304c91e-b6e4-41b5-b11c-81d4f37daa63 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:20.501812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec22435-a6b8-45cb-b1fc-eedf0394d1bc None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:20.528297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:20.542426 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 319/634] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:05:20.546211 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12afa7fd-f980-41e1-9c5c-0c393005e743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:20.547129 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12afa7fd-f980-41e1-9c5c-0c393005e743 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:20.547777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12afa7fd-f980-41e1-9c5c-0c393005e743 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:20.547895 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1304c91e-b6e4-41b5-b11c-81d4f37daa63 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:20.548395 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12afa7fd-f980-41e1-9c5c-0c393005e743 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:20.549284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12afa7fd-f980-41e1-9c5c-0c393005e743 None admin] RBAC: Adding query filter params (name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:20.549822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12afa7fd-f980-41e1-9c5c-0c393005e743 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:20.553333 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12afa7fd-f980-41e1-9c5c-0c393005e743 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:20.561405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:20.567699 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 316/635] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:20.569270 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a7ff97-595c-4563-a21b-47de1d26ea7a None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:20.569556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a7ff97-595c-4563-a21b-47de1d26ea7a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:20.569855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a7ff97-595c-4563-a21b-47de1d26ea7a None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:20.571402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 317/636] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:20.572130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a7ff97-595c-4563-a21b-47de1d26ea7a None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:20.667772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1304c91e-b6e4-41b5-b11c-81d4f37daa63 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:20.691282 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a7ff97-595c-4563-a21b-47de1d26ea7a None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:20.704182 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fab7cb6-b419-4b24-a129-4dc32629450a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:20.704824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fab7cb6-b419-4b24-a129-4dc32629450a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:20.705097 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fab7cb6-b419-4b24-a129-4dc32629450a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:20.705365 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fab7cb6-b419-4b24-a129-4dc32629450a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:20.716041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fab7cb6-b419-4b24-a129-4dc32629450a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:20.720775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-2fab7cb6-b419-4b24-a129-4dc32629450a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:20.720775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:20.720775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:20.720775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:20.720775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:20.720775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:20.720775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:20.720775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:20.720775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:20.720775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:20.720775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:20.720775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:20.720775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:20.720775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:20.720775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:20.720775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:20.721756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:20.721756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:20.721756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:20.721756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:20.721756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:20.721756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:20.721756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:20.721756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:20.721756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:20.721756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:20.721756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:20.721756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:20.721756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:20.721756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:20.721756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:20.721756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:20.722651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:20.722651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:20.722651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:20.722651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:20.722651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:20.722651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:20.722651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:20.722651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:20.722651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:20.722651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:20.722651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:20.722651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:20.722651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 320/637] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:20.722651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fab7cb6-b419-4b24-a129-4dc32629450a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:20.734132 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71841ff2-06f1-4ead-a91d-ee686dec9b27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:20.734886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71841ff2-06f1-4ead-a91d-ee686dec9b27 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:20.735184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71841ff2-06f1-4ead-a91d-ee686dec9b27 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:20.735435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71841ff2-06f1-4ead-a91d-ee686dec9b27 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:20.747322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71841ff2-06f1-4ead-a91d-ee686dec9b27 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:20.748804 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71841ff2-06f1-4ead-a91d-ee686dec9b27 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:20.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-71841ff2-06f1-4ead-a91d-ee686dec9b27 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:20.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:20.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:20.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:20.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:20.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:20.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:20.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:20.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:20.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:20.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:20.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:20.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:20.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:20.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:20.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:20.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:20.772038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:20.772038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:20.772038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:20.772038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 318/638] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:20.772038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342ba824-ac31-4a7f-a98b-94ad84eacbfa None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:20.772038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342ba824-ac31-4a7f-a98b-94ad84eacbfa None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:20.772038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342ba824-ac31-4a7f-a98b-94ad84eacbfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:20.788556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-342ba824-ac31-4a7f-a98b-94ad84eacbfa None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:20.789862 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-342ba824-ac31-4a7f-a98b-94ad84eacbfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:20.807512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:20.822834 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:20.841582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fab7cb6-b419-4b24-a129-4dc32629450a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:20.861189 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 319/639] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:20.862331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342ba824-ac31-4a7f-a98b-94ad84eacbfa None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:20.898702 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b12c8a-bbba-434c-9912-c5986ffc2cba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:20.900301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b12c8a-bbba-434c-9912-c5986ffc2cba None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:20.900729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b12c8a-bbba-434c-9912-c5986ffc2cba None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:20.901215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b12c8a-bbba-434c-9912-c5986ffc2cba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:20.915598 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b12c8a-bbba-434c-9912-c5986ffc2cba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:20.918046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b12c8a-bbba-434c-9912-c5986ffc2cba None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:20.939050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-66b12c8a-bbba-434c-9912-c5986ffc2cba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:20.939050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:20.939050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:20.939050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:20.939050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:20.939050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:20.939050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:20.939050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:20.939050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:20.939050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:20.939050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:20.939050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:20.939050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:20.939050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:20.939050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:20.939050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:20.940274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:20.941874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:20.941874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:20.941874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:20.949344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 321/640] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:20.949344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b12c8a-bbba-434c-9912-c5986ffc2cba None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:20.982609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342ba824-ac31-4a7f-a98b-94ad84eacbfa None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:21.025647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffa7c2a-2872-4e65-a872-a4dc4984b6c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:21.027044 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffa7c2a-2872-4e65-a872-a4dc4984b6c8 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:21.027612 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffa7c2a-2872-4e65-a872-a4dc4984b6c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:21.028031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffa7c2a-2872-4e65-a872-a4dc4984b6c8 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:21.029574 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffa7c2a-2872-4e65-a872-a4dc4984b6c8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:21.030040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffa7c2a-2872-4e65-a872-a4dc4984b6c8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:21.032079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffa7c2a-2872-4e65-a872-a4dc4984b6c8 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:21.044531 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 320/641] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:21.045630 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffa7c2a-2872-4e65-a872-a4dc4984b6c8 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:21.078951 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b12c8a-bbba-434c-9912-c5986ffc2cba None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:21.083018 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:21.105000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:21.117307 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0a0b61-ff8b-444e-9d3c-7cab29afd962 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:21.117963 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0a0b61-ff8b-444e-9d3c-7cab29afd962 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:21.118242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0a0b61-ff8b-444e-9d3c-7cab29afd962 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:21.118517 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0a0b61-ff8b-444e-9d3c-7cab29afd962 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:21.119211 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0a0b61-ff8b-444e-9d3c-7cab29afd962 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:21.119515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0a0b61-ff8b-444e-9d3c-7cab29afd962 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:21.121090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0a0b61-ff8b-444e-9d3c-7cab29afd962 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:21.139841 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 322/642] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:21.139841 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0a0b61-ff8b-444e-9d3c-7cab29afd962 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:21.190267 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffa7c2a-2872-4e65-a872-a4dc4984b6c8 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:21.227638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f1d100-6086-487d-9972-a2eb7d17420d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:21.228390 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f1d100-6086-487d-9972-a2eb7d17420d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:21.228670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f1d100-6086-487d-9972-a2eb7d17420d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:21.229061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f1d100-6086-487d-9972-a2eb7d17420d None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:21.229696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f1d100-6086-487d-9972-a2eb7d17420d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:21.229999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f1d100-6086-487d-9972-a2eb7d17420d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:21.231726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f1d100-6086-487d-9972-a2eb7d17420d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:21.247958 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 321/643] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:21.247958 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f1d100-6086-487d-9972-a2eb7d17420d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:21.252131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0a0b61-ff8b-444e-9d3c-7cab29afd962 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:21.291280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f915b2-36ff-4123-aa7f-1ed2d05ce44c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:21.292142 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f915b2-36ff-4123-aa7f-1ed2d05ce44c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:21.292349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f915b2-36ff-4123-aa7f-1ed2d05ce44c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:21.292610 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f915b2-36ff-4123-aa7f-1ed2d05ce44c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:21.305279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f915b2-36ff-4123-aa7f-1ed2d05ce44c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:21.306731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f915b2-36ff-4123-aa7f-1ed2d05ce44c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:21.322736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-84f915b2-36ff-4123-aa7f-1ed2d05ce44c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:21.322736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:21.322736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:21.322736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:21.322736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:21.322736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:21.322736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:21.322736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:21.322736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:21.322736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:21.322736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:21.322736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:21.322736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:21.322736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:21.322736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:21.322736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:21.323843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:21.325071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:21.325071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:21.325071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:21.328563 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 323/644] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:21.329386 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f915b2-36ff-4123-aa7f-1ed2d05ce44c None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:21.337309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:21.360008 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f1d100-6086-487d-9972-a2eb7d17420d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:21.367852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:21.397864 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c63e07d-7693-4aa9-9d60-324cdf68f075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:21.398536 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c63e07d-7693-4aa9-9d60-324cdf68f075 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:21.398837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c63e07d-7693-4aa9-9d60-324cdf68f075 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:21.399112 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c63e07d-7693-4aa9-9d60-324cdf68f075 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:21.406246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c63e07d-7693-4aa9-9d60-324cdf68f075 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:21.407696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c63e07d-7693-4aa9-9d60-324cdf68f075 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:21.421340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-5c63e07d-7693-4aa9-9d60-324cdf68f075 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:21.421340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:21.421340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:21.421340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:21.421340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:21.421340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:21.421340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:21.421340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:21.421340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:21.421340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:21.421340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:21.421340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:21.421340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:21.421340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:21.421340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:21.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:21.423556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:21.423556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:21.423556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:21.423556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:21.423556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:21.423556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:21.423556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:21.423556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:21.423556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 322/645] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:21.423556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c63e07d-7693-4aa9-9d60-324cdf68f075 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:21.449737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f915b2-36ff-4123-aa7f-1ed2d05ce44c None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:21.488910 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfcc334-42d6-4967-a303-722641678f86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:21.489662 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfcc334-42d6-4967-a303-722641678f86 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:21.490001 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfcc334-42d6-4967-a303-722641678f86 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:21.490341 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfcc334-42d6-4967-a303-722641678f86 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:21.501683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfcc334-42d6-4967-a303-722641678f86 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:21.507699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-4bfcc334-42d6-4967-a303-722641678f86 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:21.507699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:21.507699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:21.507699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:21.507699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:21.507699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:21.507699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:21.507699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:21.507699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:21.507699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:21.507699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:21.507699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:21.507699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:21.507699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:21.507699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:21.507699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:21.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:21.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:21.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:21.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:21.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:21.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:21.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:21.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:21.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:21.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:21.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:21.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:21.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:21.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:21.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:21.508660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:21.513176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:21.513176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:21.513176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:21.513176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:21.513176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:21.513176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:21.513176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:21.513176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:21.513176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:21.513176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:21.513176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:21.513176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:21.513176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 324/646] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:21.513176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfcc334-42d6-4967-a303-722641678f86 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:21.539026 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c63e07d-7693-4aa9-9d60-324cdf68f075 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:21.574493 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed1063a-fb02-47b8-a094-81769339b0cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:21.575147 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed1063a-fb02-47b8-a094-81769339b0cc None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:21.575421 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed1063a-fb02-47b8-a094-81769339b0cc None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:21.575689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed1063a-fb02-47b8-a094-81769339b0cc None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:21.583520 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed1063a-fb02-47b8-a094-81769339b0cc None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:21.584906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed1063a-fb02-47b8-a094-81769339b0cc None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:21.596177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-3ed1063a-fb02-47b8-a094-81769339b0cc None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 21 01:05:21.596177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:21.596177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:21.596177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:21.596177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:21.596177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:21.596177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:21.596177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:21.596177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:21.596177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:21.596177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:21.596177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:21.596177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:21.596177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:21.596177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:21.596177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:21.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:21.598739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:21.598739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:21.598739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:21.598739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:21.598739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:21.598739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:21.598739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:21.598739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:21.598739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 21 01:05:21.598739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:21.598739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 323/647] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:05:21 2022] GET /identity/v3/users/cinder => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:21.598739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed1063a-fb02-47b8-a094-81769339b0cc None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:21.603670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:21.619232 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:21.625940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfcc334-42d6-4967-a303-722641678f86 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:21.665171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb2c6e2-abca-40cc-bd2c-193c8185f1d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:21.665836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb2c6e2-abca-40cc-bd2c-193c8185f1d6 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:21.666034 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb2c6e2-abca-40cc-bd2c-193c8185f1d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:21.666313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb2c6e2-abca-40cc-bd2c-193c8185f1d6 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:21.666981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb2c6e2-abca-40cc-bd2c-193c8185f1d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:21.667398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb2c6e2-abca-40cc-bd2c-193c8185f1d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:21.668865 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb2c6e2-abca-40cc-bd2c-193c8185f1d6 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:21.686599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 325/648] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:21.686599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb2c6e2-abca-40cc-bd2c-193c8185f1d6 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:21.712341 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed1063a-fb02-47b8-a094-81769339b0cc None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:21.747875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a7682a-f801-414d-8b2c-7582d5170e97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:21.748520 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a7682a-f801-414d-8b2c-7582d5170e97 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:21.748798 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a7682a-f801-414d-8b2c-7582d5170e97 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:21.749061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a7682a-f801-414d-8b2c-7582d5170e97 None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:21.749749 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a7682a-f801-414d-8b2c-7582d5170e97 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:21.750055 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a7682a-f801-414d-8b2c-7582d5170e97 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:21.751472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a7682a-f801-414d-8b2c-7582d5170e97 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:21.763392 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 324/649] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:21.764247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a7682a-f801-414d-8b2c-7582d5170e97 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:21.794790 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb2c6e2-abca-40cc-bd2c-193c8185f1d6 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:21.830113 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfb5410-3c1f-418f-92eb-f6f8f86b2e50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:21.830726 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfb5410-3c1f-418f-92eb-f6f8f86b2e50 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:21.830999 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfb5410-3c1f-418f-92eb-f6f8f86b2e50 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:21.831274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfb5410-3c1f-418f-92eb-f6f8f86b2e50 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:21.831961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfb5410-3c1f-418f-92eb-f6f8f86b2e50 None admin] RBAC: Adding query filter params (name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:21.832311 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfb5410-3c1f-418f-92eb-f6f8f86b2e50 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:21.833740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfb5410-3c1f-418f-92eb-f6f8f86b2e50 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:21.838510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:21.847246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 326/650] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:21.847246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfb5410-3c1f-418f-92eb-f6f8f86b2e50 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:21.865803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:21.883356 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a7682a-f801-414d-8b2c-7582d5170e97 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:21.918260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d54f41a-1a53-46b1-bf73-6a6a73fc5085 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:21.918958 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d54f41a-1a53-46b1-bf73-6a6a73fc5085 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:21.919240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d54f41a-1a53-46b1-bf73-6a6a73fc5085 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:21.919506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d54f41a-1a53-46b1-bf73-6a6a73fc5085 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:21.920264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d54f41a-1a53-46b1-bf73-6a6a73fc5085 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:21.920573 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d54f41a-1a53-46b1-bf73-6a6a73fc5085 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:21.921983 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d54f41a-1a53-46b1-bf73-6a6a73fc5085 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:21.940163 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 325/651] 10.222.0.47 () {56 vars in 1137 bytes} [Sun Aug 21 01:05:21 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:21.940994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d54f41a-1a53-46b1-bf73-6a6a73fc5085 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:21.959098 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfb5410-3c1f-418f-92eb-f6f8f86b2e50 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:21.996242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6410b98-72cf-4b81-b16e-634dcfe51b3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:21.997157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6410b98-72cf-4b81-b16e-634dcfe51b3a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:21.997425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6410b98-72cf-4b81-b16e-634dcfe51b3a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:21.997692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6410b98-72cf-4b81-b16e-634dcfe51b3a None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:22.005184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6410b98-72cf-4b81-b16e-634dcfe51b3a None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:22.006606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6410b98-72cf-4b81-b16e-634dcfe51b3a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:22.017183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-a6410b98-72cf-4b81-b16e-634dcfe51b3a None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 21 01:05:22.017183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:22.017183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:22.017183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:22.017183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:22.017183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:22.017183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:22.017183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:22.017183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:22.017183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:22.017183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:22.017183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:22.017183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:22.017183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:22.017183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:22.017183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:22.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:22.019124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:22.019124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:22.019124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:22.019124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:22.019124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:22.019124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:22.019124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:22.019124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:22.019124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 21 01:05:22.019124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:22.019124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 327/652] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:21 2022] GET /identity/v3/users/neutron => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:22.019124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6410b98-72cf-4b81-b16e-634dcfe51b3a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:22.062720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d54f41a-1a53-46b1-bf73-6a6a73fc5085 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:22.088135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:22.101508 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26065284-afd3-4117-a556-eb2f0d75cdb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:22.102262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26065284-afd3-4117-a556-eb2f0d75cdb4 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:22.102513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26065284-afd3-4117-a556-eb2f0d75cdb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:22.102833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26065284-afd3-4117-a556-eb2f0d75cdb4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:22.103978 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26065284-afd3-4117-a556-eb2f0d75cdb4 None admin] RBAC: Adding query filter params (role.id=c2d8bf7283e14857879d03d1f39c0165, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=fabdcdc25d1d4ef6bc66bd6cadecaaf8) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:22.104324 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26065284-afd3-4117-a556-eb2f0d75cdb4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:22.105616 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:22.106129 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26065284-afd3-4117-a556-eb2f0d75cdb4 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:22.113670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 326/653] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:21 2022] GET /identity/v3/role_assignments?user.id=fabdcdc25d1d4ef6bc66bd6cadecaaf8&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=c2d8bf7283e14857879d03d1f39c0165 => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:22.114223 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26065284-afd3-4117-a556-eb2f0d75cdb4 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:22.153989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6410b98-72cf-4b81-b16e-634dcfe51b3a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:22.208445 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46823490-1339-4253-a3d5-000a3b2f4e92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:22.209307 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46823490-1339-4253-a3d5-000a3b2f4e92 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:22.209736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46823490-1339-4253-a3d5-000a3b2f4e92 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:22.210806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46823490-1339-4253-a3d5-000a3b2f4e92 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:22.227596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46823490-1339-4253-a3d5-000a3b2f4e92 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:22.229572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46823490-1339-4253-a3d5-000a3b2f4e92 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:22.241244 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26065284-afd3-4117-a556-eb2f0d75cdb4 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:22.251126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-46823490-1339-4253-a3d5-000a3b2f4e92 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:22.251126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:22.251126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:22.251126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:22.251126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:22.251126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:22.251126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:22.251126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:22.251126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:22.251126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:22.251126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:22.251126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:22.251126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:22.251126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:22.251126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:22.251126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:22.252406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:22.253384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:22.253384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:22.253384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:22.256267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 328/654] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:22.257411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46823490-1339-4253-a3d5-000a3b2f4e92 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:22.288034 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7e6551-4fc8-46a1-b890-7f245ea5f076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:22.288903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7e6551-4fc8-46a1-b890-7f245ea5f076 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:22.289171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7e6551-4fc8-46a1-b890-7f245ea5f076 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:22.289466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7e6551-4fc8-46a1-b890-7f245ea5f076 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:22.354915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:22.370952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7e6551-4fc8-46a1-b890-7f245ea5f076 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:22.373503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7e6551-4fc8-46a1-b890-7f245ea5f076 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:22.382252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46823490-1339-4253-a3d5-000a3b2f4e92 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:22.409120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:22.417967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022e52b2-3bf6-48de-a4cf-61ee5c311691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:22.418681 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022e52b2-3bf6-48de-a4cf-61ee5c311691 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:22.418959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022e52b2-3bf6-48de-a4cf-61ee5c311691 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:22.419231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022e52b2-3bf6-48de-a4cf-61ee5c311691 None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:22.419954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022e52b2-3bf6-48de-a4cf-61ee5c311691 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:22.420287 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022e52b2-3bf6-48de-a4cf-61ee5c311691 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:22.421743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022e52b2-3bf6-48de-a4cf-61ee5c311691 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:22.440404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 329/655] 10.222.0.47 () {56 vars in 1139 bytes} [Sun Aug 21 01:05:22 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:22.441474 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022e52b2-3bf6-48de-a4cf-61ee5c311691 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:22.495920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 327/656] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:05:22.497226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ee6614-11d9-4128-acf7-581a2e7d2e2d None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:22.497510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ee6614-11d9-4128-acf7-581a2e7d2e2d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:22.497783 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ee6614-11d9-4128-acf7-581a2e7d2e2d None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:22.499192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 328/657] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:22.499715 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ee6614-11d9-4128-acf7-581a2e7d2e2d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:22.557450 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022e52b2-3bf6-48de-a4cf-61ee5c311691 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:22.590090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:22.595308 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b1aa7d-2b4d-4d4e-a63a-e2d923d29157 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:22.595915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b1aa7d-2b4d-4d4e-a63a-e2d923d29157 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:22.596233 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b1aa7d-2b4d-4d4e-a63a-e2d923d29157 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:22.596486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b1aa7d-2b4d-4d4e-a63a-e2d923d29157 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:22.597116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b1aa7d-2b4d-4d4e-a63a-e2d923d29157 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:22.598155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b1aa7d-2b4d-4d4e-a63a-e2d923d29157 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:22.602808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b1aa7d-2b4d-4d4e-a63a-e2d923d29157 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:22.607678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ee6614-11d9-4128-acf7-581a2e7d2e2d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:22.617367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 330/658] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:22.617367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b1aa7d-2b4d-4d4e-a63a-e2d923d29157 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:22.643454 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f2785f-fa40-45c6-9932-6fea09cb9b66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:22.644220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f2785f-fa40-45c6-9932-6fea09cb9b66 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:22.644680 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f2785f-fa40-45c6-9932-6fea09cb9b66 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:22.645026 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f2785f-fa40-45c6-9932-6fea09cb9b66 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:22.670150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:22.720286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f2785f-fa40-45c6-9932-6fea09cb9b66 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:22.723150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f2785f-fa40-45c6-9932-6fea09cb9b66 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:22.728404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b1aa7d-2b4d-4d4e-a63a-e2d923d29157 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:22.763877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700258f5-21a6-480e-a978-7ba831771cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:22.764535 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700258f5-21a6-480e-a978-7ba831771cfc None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:22.764809 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700258f5-21a6-480e-a978-7ba831771cfc None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:22.765076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700258f5-21a6-480e-a978-7ba831771cfc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:22.778351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700258f5-21a6-480e-a978-7ba831771cfc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:22.780361 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700258f5-21a6-480e-a978-7ba831771cfc None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:22.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-700258f5-21a6-480e-a978-7ba831771cfc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:22.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:22.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:22.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:22.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:22.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:22.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:22.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:22.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:22.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:22.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:22.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:22.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:22.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:22.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:22.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:22.799104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:22.800640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:22.800640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:22.800640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:22.807589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 331/659] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:22.809743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e024d47e-b15b-4e0a-8f57-17116494604f None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:22.810025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e024d47e-b15b-4e0a-8f57-17116494604f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:22.810325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e024d47e-b15b-4e0a-8f57-17116494604f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:22.823524 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:22.828593 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-e024d47e-b15b-4e0a-8f57-17116494604f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:22.829796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e024d47e-b15b-4e0a-8f57-17116494604f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:22.840669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 329/660] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:05:22.841635 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f2785f-fa40-45c6-9932-6fea09cb9b66 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:22.906982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 332/661] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:22.908035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e024d47e-b15b-4e0a-8f57-17116494604f None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:22.937581 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:22.949648 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f2785f-fa40-45c6-9932-6fea09cb9b66 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:22.987283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9073b3-8886-43da-9460-d9a8d675fad2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:22.988356 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9073b3-8886-43da-9460-d9a8d675fad2 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:22.988816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9073b3-8886-43da-9460-d9a8d675fad2 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:22.989263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9073b3-8886-43da-9460-d9a8d675fad2 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:22.997259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9073b3-8886-43da-9460-d9a8d675fad2 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:22.998903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9073b3-8886-43da-9460-d9a8d675fad2 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:23.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-1b9073b3-8886-43da-9460-d9a8d675fad2 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 21 01:05:23.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:23.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:23.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:23.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:23.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:23.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:23.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:23.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:23.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:23.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:23.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:23.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:23.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:23.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:23.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:23.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:23.013602 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:23.013602 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:23.013602 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:23.013602 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:23.013602 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:23.013602 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:23.013602 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:23.013602 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:23.013602 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 21 01:05:23.013602 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:23.014154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e024d47e-b15b-4e0a-8f57-17116494604f None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:23.020823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 330/662] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:05:22 2022] GET /identity/v3/users/glance => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:23.021935 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9073b3-8886-43da-9460-d9a8d675fad2 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:23.048065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98acdcb0-9986-4334-a8cf-2193e6264809 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:23.048715 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98acdcb0-9986-4334-a8cf-2193e6264809 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:23.048992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98acdcb0-9986-4334-a8cf-2193e6264809 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:23.049262 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98acdcb0-9986-4334-a8cf-2193e6264809 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:23.049959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98acdcb0-9986-4334-a8cf-2193e6264809 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:23.050204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98acdcb0-9986-4334-a8cf-2193e6264809 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:23.051607 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98acdcb0-9986-4334-a8cf-2193e6264809 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:23.067855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 333/663] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:23.067855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98acdcb0-9986-4334-a8cf-2193e6264809 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:23.080384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:23.140815 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9073b3-8886-43da-9460-d9a8d675fad2 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:23.156798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:23.169313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98acdcb0-9986-4334-a8cf-2193e6264809 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:23.180404 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fc4cad-2c03-46c5-acad-06336dd1d6af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:23.181248 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fc4cad-2c03-46c5-acad-06336dd1d6af None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:23.181795 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fc4cad-2c03-46c5-acad-06336dd1d6af None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:23.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fc4cad-2c03-46c5-acad-06336dd1d6af None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:23.194891 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fc4cad-2c03-46c5-acad-06336dd1d6af None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:23.196384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fc4cad-2c03-46c5-acad-06336dd1d6af None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:23.205426 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ce8f31-049d-47dd-b646-696a98d9fbc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:23.206353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ce8f31-049d-47dd-b646-696a98d9fbc5 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:23.206634 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ce8f31-049d-47dd-b646-696a98d9fbc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:23.207463 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ce8f31-049d-47dd-b646-696a98d9fbc5 None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:23.208221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ce8f31-049d-47dd-b646-696a98d9fbc5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:23.208526 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ce8f31-049d-47dd-b646-696a98d9fbc5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:23.209956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ce8f31-049d-47dd-b646-696a98d9fbc5 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:23.211277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-11fc4cad-2c03-46c5-acad-06336dd1d6af None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:23.211277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:23.211277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:23.211277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:23.211277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:23.211277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:23.211277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:23.211277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:23.211277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:23.211277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:23.211277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:23.211277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:23.211277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:23.211277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:23.211277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:23.211277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:23.212240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:23.213192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:23.213192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:23.213192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:23.213192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 331/664] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:23.213708 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fc4cad-2c03-46c5-acad-06336dd1d6af None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:23.236228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 334/665] 10.222.0.47 () {56 vars in 1137 bytes} [Sun Aug 21 01:05:23 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:23.236228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ce8f31-049d-47dd-b646-696a98d9fbc5 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:23.329390 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:23.334089 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fc4cad-2c03-46c5-acad-06336dd1d6af None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:23.345443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ce8f31-049d-47dd-b646-696a98d9fbc5 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:23.375180 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a36ac8-fd97-4a70-b2c7-8d9fad6e9ab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:23.375957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a36ac8-fd97-4a70-b2c7-8d9fad6e9ab5 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:23.376354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a36ac8-fd97-4a70-b2c7-8d9fad6e9ab5 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:23.376693 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a36ac8-fd97-4a70-b2c7-8d9fad6e9ab5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:23.383641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a36ac8-fd97-4a70-b2c7-8d9fad6e9ab5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:23.383848 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822840d7-a0e8-4af0-b0c1-ecada761660b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:23.384533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822840d7-a0e8-4af0-b0c1-ecada761660b None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:23.384878 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822840d7-a0e8-4af0-b0c1-ecada761660b None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:23.385215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822840d7-a0e8-4af0-b0c1-ecada761660b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:23.385456 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a36ac8-fd97-4a70-b2c7-8d9fad6e9ab5 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:23.396206 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822840d7-a0e8-4af0-b0c1-ecada761660b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:23.398247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-e3a36ac8-fd97-4a70-b2c7-8d9fad6e9ab5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:23.398247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:23.398247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:23.398247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:23.398247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:23.398247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:23.398247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:23.398247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:23.398247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:23.398247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:23.398247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:23.398247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:23.398247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:23.398247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:23.398247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:23.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:23.400458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:23.400458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:23.400458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:23.400458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:23.400458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:23.400458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:23.400458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:23.400458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:23.401345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 332/666] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:23.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-822840d7-a0e8-4af0-b0c1-ecada761660b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:23.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:23.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:23.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:23.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:23.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:23.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:23.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:23.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:23.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:23.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:23.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:23.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:23.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:23.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:23.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:23.404198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:23.404198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:23.404198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:23.404198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:23.404198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:23.404198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:23.404198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:23.404198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:23.404198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:23.404198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:23.404198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:23.404198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:23.404198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:23.404198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:23.404198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:23.404198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:23.405342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:23.405342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:23.405342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:23.405342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:23.405342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:23.405342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:23.405342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:23.405342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:23.405342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:23.405342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:23.405342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:23.405342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:23.405342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 335/667] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:23 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:23.407251 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a36ac8-fd97-4a70-b2c7-8d9fad6e9ab5 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:23.411872 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822840d7-a0e8-4af0-b0c1-ecada761660b None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:23.421387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:23.518900 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a36ac8-fd97-4a70-b2c7-8d9fad6e9ab5 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:23.520024 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822840d7-a0e8-4af0-b0c1-ecada761660b None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:23.556571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb97d83e-289a-4643-a9c6-609d169371bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:23.557012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ede03e-eb31-4c59-9f4e-5f567211e39c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:23.557441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb97d83e-289a-4643-a9c6-609d169371bb None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:23.557619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ede03e-eb31-4c59-9f4e-5f567211e39c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:23.557896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ede03e-eb31-4c59-9f4e-5f567211e39c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:23.557989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb97d83e-289a-4643-a9c6-609d169371bb None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:23.558166 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ede03e-eb31-4c59-9f4e-5f567211e39c None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:23.558402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb97d83e-289a-4643-a9c6-609d169371bb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:23.558803 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ede03e-eb31-4c59-9f4e-5f567211e39c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:23.559108 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ede03e-eb31-4c59-9f4e-5f567211e39c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:23.560565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ede03e-eb31-4c59-9f4e-5f567211e39c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:23.573768 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 333/668] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:23.574576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ede03e-eb31-4c59-9f4e-5f567211e39c None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:23.595163 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:23.635984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb97d83e-289a-4643-a9c6-609d169371bb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:23.637560 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb97d83e-289a-4643-a9c6-609d169371bb None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:23.646992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:23.680205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ede03e-eb31-4c59-9f4e-5f567211e39c None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:23.720394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fbea26-2efe-44f0-bd67-b5558e7aa6ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:23.721007 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fbea26-2efe-44f0-bd67-b5558e7aa6ea None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:23.721287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fbea26-2efe-44f0-bd67-b5558e7aa6ea None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:23.721562 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fbea26-2efe-44f0-bd67-b5558e7aa6ea None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:23.722991 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fbea26-2efe-44f0-bd67-b5558e7aa6ea None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:23.723306 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fbea26-2efe-44f0-bd67-b5558e7aa6ea None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:23.725591 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fbea26-2efe-44f0-bd67-b5558e7aa6ea None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:23.738875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 334/669] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:23.739706 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fbea26-2efe-44f0-bd67-b5558e7aa6ea None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:23.767231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 336/670] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:05:23.767231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb97d83e-289a-4643-a9c6-609d169371bb None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:23.837459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:23.859359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fbea26-2efe-44f0-bd67-b5558e7aa6ea None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:23.876122 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb97d83e-289a-4643-a9c6-609d169371bb None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:23.884308 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:23.896700 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0d4eb4-d125-4c52-8b14-4126bcfcd10f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:23.897394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0d4eb4-d125-4c52-8b14-4126bcfcd10f None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:23.897746 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0d4eb4-d125-4c52-8b14-4126bcfcd10f None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:23.898079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0d4eb4-d125-4c52-8b14-4126bcfcd10f None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:23.898744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0d4eb4-d125-4c52-8b14-4126bcfcd10f None admin] RBAC: Adding query filter params (name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:23.899044 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0d4eb4-d125-4c52-8b14-4126bcfcd10f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:23.900423 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0d4eb4-d125-4c52-8b14-4126bcfcd10f None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:23.909165 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 335/671] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:23.909819 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0d4eb4-d125-4c52-8b14-4126bcfcd10f None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:23.917037 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75343246-ebb7-4f7b-bd96-cd4a41c8af57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:23.917751 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75343246-ebb7-4f7b-bd96-cd4a41c8af57 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:23.918098 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75343246-ebb7-4f7b-bd96-cd4a41c8af57 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:23.918438 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75343246-ebb7-4f7b-bd96-cd4a41c8af57 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:23.927249 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75343246-ebb7-4f7b-bd96-cd4a41c8af57 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:23.929513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75343246-ebb7-4f7b-bd96-cd4a41c8af57 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:23.945642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-75343246-ebb7-4f7b-bd96-cd4a41c8af57 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:23.945642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:23.945642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:23.945642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:23.945642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:23.945642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:23.945642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:23.945642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:23.945642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:23.945642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:23.945642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:23.945642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:23.945642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:23.945642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:23.945642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:23.946529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:23.947414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:23.947414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:23.947414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:23.947414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:23.947414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:23.947414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:23.947414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:23.947414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:23.947414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 337/672] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:23.947414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75343246-ebb7-4f7b-bd96-cd4a41c8af57 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:24.026604 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0d4eb4-d125-4c52-8b14-4126bcfcd10f None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:24.061987 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75343246-ebb7-4f7b-bd96-cd4a41c8af57 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:24.063364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdd6623-220b-4b7a-96d1-e5417bd20a82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:24.063972 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdd6623-220b-4b7a-96d1-e5417bd20a82 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:24.064283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdd6623-220b-4b7a-96d1-e5417bd20a82 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:24.064560 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdd6623-220b-4b7a-96d1-e5417bd20a82 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:24.065281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdd6623-220b-4b7a-96d1-e5417bd20a82 None admin] RBAC: Adding query filter params (role.id=c2d8bf7283e14857879d03d1f39c0165, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=b953036b8b774f25bda59cec96b21c27) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:24.065585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdd6623-220b-4b7a-96d1-e5417bd20a82 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:24.067798 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdd6623-220b-4b7a-96d1-e5417bd20a82 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:24.073781 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 336/673] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:23 2022] GET /identity/v3/role_assignments?user.id=b953036b8b774f25bda59cec96b21c27&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=c2d8bf7283e14857879d03d1f39c0165 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:24.074572 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdd6623-220b-4b7a-96d1-e5417bd20a82 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:24.078993 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:24.126492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eaa150d-c1a9-44d1-87e2-65adb2a18adf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:24.127138 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaa150d-c1a9-44d1-87e2-65adb2a18adf None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:24.127423 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaa150d-c1a9-44d1-87e2-65adb2a18adf None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:24.127690 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaa150d-c1a9-44d1-87e2-65adb2a18adf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:24.139656 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eaa150d-c1a9-44d1-87e2-65adb2a18adf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:24.141381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eaa150d-c1a9-44d1-87e2-65adb2a18adf None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:24.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-0eaa150d-c1a9-44d1-87e2-65adb2a18adf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:24.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:24.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:24.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:24.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:24.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:24.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:24.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:24.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:24.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:24.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:24.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:24.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:24.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:24.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:24.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:24.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:24.160787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:24.160787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:24.160787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:24.160787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 338/674] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:24.161026 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eaa150d-c1a9-44d1-87e2-65adb2a18adf None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:24.182459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:24.204465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdd6623-220b-4b7a-96d1-e5417bd20a82 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:24.247257 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bcddd64-a245-43ca-ab93-914f224e6bcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:24.247886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcddd64-a245-43ca-ab93-914f224e6bcf None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:24.248239 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcddd64-a245-43ca-ab93-914f224e6bcf None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:24.248476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcddd64-a245-43ca-ab93-914f224e6bcf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:24.260934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bcddd64-a245-43ca-ab93-914f224e6bcf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:24.262447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bcddd64-a245-43ca-ab93-914f224e6bcf None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:24.280196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-8bcddd64-a245-43ca-ab93-914f224e6bcf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:24.280196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:24.280196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:24.280196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:24.280196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:24.280196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:24.280196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:24.280196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:24.280196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:24.280196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:24.280196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:24.280196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:24.280196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:24.280196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:24.280196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:24.280196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:24.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:24.282100 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:24.282100 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:24.282100 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:24.282272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eaa150d-c1a9-44d1-87e2-65adb2a18adf None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:24.282346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 337/675] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:24.282886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7700759-8b78-4815-ae09-d03f15bbdbc9 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:24.283052 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7700759-8b78-4815-ae09-d03f15bbdbc9 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:24.283312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7700759-8b78-4815-ae09-d03f15bbdbc9 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:24.284900 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 338/676] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:24.286081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7700759-8b78-4815-ae09-d03f15bbdbc9 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:24.320362 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdc656d-6876-45af-a6f9-2130654c79fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:24.321400 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdc656d-6876-45af-a6f9-2130654c79fc None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:24.321886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdc656d-6876-45af-a6f9-2130654c79fc None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:24.322162 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdc656d-6876-45af-a6f9-2130654c79fc None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:24.322854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdc656d-6876-45af-a6f9-2130654c79fc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:24.323155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdc656d-6876-45af-a6f9-2130654c79fc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:24.324741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdc656d-6876-45af-a6f9-2130654c79fc None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:24.337603 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 339/677] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:24.343925 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdc656d-6876-45af-a6f9-2130654c79fc None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:24.365245 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:24.413761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7700759-8b78-4815-ae09-d03f15bbdbc9 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:24.430927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:24.453155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdc656d-6876-45af-a6f9-2130654c79fc None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:24.456479 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37807dae-2ae6-4f7c-a60f-0bf0df78cb17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:24.457816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37807dae-2ae6-4f7c-a60f-0bf0df78cb17 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:24.458325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37807dae-2ae6-4f7c-a60f-0bf0df78cb17 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:24.458775 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37807dae-2ae6-4f7c-a60f-0bf0df78cb17 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:24.459591 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37807dae-2ae6-4f7c-a60f-0bf0df78cb17 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:24.460059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37807dae-2ae6-4f7c-a60f-0bf0df78cb17 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:24.462113 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37807dae-2ae6-4f7c-a60f-0bf0df78cb17 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:24.477036 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 339/678] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:24.478273 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11662ef-3110-4df7-a3ee-5fc85a00147d None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:24.478601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11662ef-3110-4df7-a3ee-5fc85a00147d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:24.478817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11662ef-3110-4df7-a3ee-5fc85a00147d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:24.490496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c09cf6-b014-4d9f-a23c-d06b3977abad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:24.491404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c09cf6-b014-4d9f-a23c-d06b3977abad None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:24.491864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c09cf6-b014-4d9f-a23c-d06b3977abad None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:24.492165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c09cf6-b014-4d9f-a23c-d06b3977abad None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:24.492806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c09cf6-b014-4d9f-a23c-d06b3977abad None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:24.493160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c09cf6-b014-4d9f-a23c-d06b3977abad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:24.494512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c09cf6-b014-4d9f-a23c-d06b3977abad None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:24.497818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f11662ef-3110-4df7-a3ee-5fc85a00147d None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:24.499018 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f11662ef-3110-4df7-a3ee-5fc85a00147d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:24.511474 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 340/679] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:24.511474 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c09cf6-b014-4d9f-a23c-d06b3977abad None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:24.582421 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 340/680] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:24.583432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11662ef-3110-4df7-a3ee-5fc85a00147d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:24.635946 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c09cf6-b014-4d9f-a23c-d06b3977abad None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:24.662348 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:24.686487 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e909a9-4ee3-4f38-ae75-25a8eae4e3ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:24.687749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e909a9-4ee3-4f38-ae75-25a8eae4e3ac None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:24.688292 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e909a9-4ee3-4f38-ae75-25a8eae4e3ac None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:24.688872 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e909a9-4ee3-4f38-ae75-25a8eae4e3ac None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:24.690550 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e909a9-4ee3-4f38-ae75-25a8eae4e3ac None admin] RBAC: Adding query filter params (role.id=c2d8bf7283e14857879d03d1f39c0165, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=b532ddab1f5e467989edaa9e4b74bd72) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:24.691086 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e909a9-4ee3-4f38-ae75-25a8eae4e3ac None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:24.692991 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e909a9-4ee3-4f38-ae75-25a8eae4e3ac None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:24.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 341/681] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:24 2022] GET /identity/v3/role_assignments?user.id=b532ddab1f5e467989edaa9e4b74bd72&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=c2d8bf7283e14857879d03d1f39c0165 => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:24.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e909a9-4ee3-4f38-ae75-25a8eae4e3ac None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:24.711081 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:24.746428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11662ef-3110-4df7-a3ee-5fc85a00147d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:24.820017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a346d1dc-3c6d-4a07-b3ed-6e3196cbb007 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:24.821402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a346d1dc-3c6d-4a07-b3ed-6e3196cbb007 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:24.821671 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a346d1dc-3c6d-4a07-b3ed-6e3196cbb007 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:24.821944 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a346d1dc-3c6d-4a07-b3ed-6e3196cbb007 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:24.835890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a346d1dc-3c6d-4a07-b3ed-6e3196cbb007 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:24.838705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a346d1dc-3c6d-4a07-b3ed-6e3196cbb007 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:24.863563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-a346d1dc-3c6d-4a07-b3ed-6e3196cbb007 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:24.863563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:24.863563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:24.863563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:24.863563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:24.863563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:24.863563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:24.863563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:24.863563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:24.863563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:24.863563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:24.863563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:24.863563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:24.863563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:24.863563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:24.864584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:24.865481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:24.865481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:24.865481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:24.865481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:24.865481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:24.865481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:24.865481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:24.865481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:24.865481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 341/682] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:24.871930 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a346d1dc-3c6d-4a07-b3ed-6e3196cbb007 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:24.900790 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e909a9-4ee3-4f38-ae75-25a8eae4e3ac None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:24.943666 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9fbc76-8ca1-4389-ab21-8fea1ff03472 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:24.944326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9fbc76-8ca1-4389-ab21-8fea1ff03472 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:24.944609 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9fbc76-8ca1-4389-ab21-8fea1ff03472 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:24.945529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9fbc76-8ca1-4389-ab21-8fea1ff03472 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:24.952713 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9fbc76-8ca1-4389-ab21-8fea1ff03472 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:24.954154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9fbc76-8ca1-4389-ab21-8fea1ff03472 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:24.965796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-9d9fbc76-8ca1-4389-ab21-8fea1ff03472 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 21 01:05:24.965796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:24.965796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:24.965796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:24.965796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:24.965796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:24.965796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:24.965796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:24.965796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:24.965796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:24.965796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:24.965796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:24.965796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:24.965796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:24.965796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:24.965796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:24.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:24.967788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:24.967788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:24.967788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:24.967788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:24.967788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:24.967788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:24.967788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:24.967788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:24.967788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 21 01:05:24.967788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:24.967788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 342/683] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:05:24 2022] GET /identity/v3/users/swift => generated 83 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:24.967788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9fbc76-8ca1-4389-ab21-8fea1ff03472 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:25.009561 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a346d1dc-3c6d-4a07-b3ed-6e3196cbb007 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:25.020709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:25.036035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:25.059825 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0adfc6-eb85-487c-b926-9ec457e56058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:25.061057 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0adfc6-eb85-487c-b926-9ec457e56058 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:25.061553 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0adfc6-eb85-487c-b926-9ec457e56058 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:25.062082 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0adfc6-eb85-487c-b926-9ec457e56058 None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:25.062961 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0adfc6-eb85-487c-b926-9ec457e56058 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:25.063879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0adfc6-eb85-487c-b926-9ec457e56058 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:25.065761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0adfc6-eb85-487c-b926-9ec457e56058 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:25.076349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9fbc76-8ca1-4389-ab21-8fea1ff03472 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:25.082865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 342/684] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:25.084260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0adfc6-eb85-487c-b926-9ec457e56058 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:25.118364 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817fdcc3-f934-49d8-9138-0cd24429cf86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:25.119133 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817fdcc3-f934-49d8-9138-0cd24429cf86 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:25.119474 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817fdcc3-f934-49d8-9138-0cd24429cf86 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:25.119814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817fdcc3-f934-49d8-9138-0cd24429cf86 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:25.132786 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817fdcc3-f934-49d8-9138-0cd24429cf86 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:25.138350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-817fdcc3-f934-49d8-9138-0cd24429cf86 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:05:25.138350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:25.138350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:25.138350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:25.138350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:25.138350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:25.138350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:25.138350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:25.138350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:25.138350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:25.138350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:25.138350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:25.138350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:25.138350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:25.138350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:25.138350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:25.139385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:25.139385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:25.139385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:25.139385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:25.139385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:25.139385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:25.139385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:25.139385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:25.139385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:25.139385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:25.139385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:25.139385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:25.139385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:25.139385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:25.139385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:25.139385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:25.140285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:25.140285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:25.140285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:25.140285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:25.140285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:25.140285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:25.140285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:25.140285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:25.140285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:25.140285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:25.140285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:05:25.140285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:25.140285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 343/685] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:05:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:25.141010 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817fdcc3-f934-49d8-9138-0cd24429cf86 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:25.266010 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817fdcc3-f934-49d8-9138-0cd24429cf86 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:25.292669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:25.311568 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b990f7-aa07-4f41-9122-ad1b95527e8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:25.312215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b990f7-aa07-4f41-9122-ad1b95527e8e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:25.312522 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b990f7-aa07-4f41-9122-ad1b95527e8e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:25.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b990f7-aa07-4f41-9122-ad1b95527e8e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:25.313544 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b990f7-aa07-4f41-9122-ad1b95527e8e None admin] RBAC: Adding query filter params (role.id=c2d8bf7283e14857879d03d1f39c0165, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=a6365b2328f34ab9b1b73ececbf5cc30) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:25.313848 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b990f7-aa07-4f41-9122-ad1b95527e8e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:25.315291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b990f7-aa07-4f41-9122-ad1b95527e8e None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:25.328131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 344/686] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:25 2022] GET /identity/v3/role_assignments?user.id=a6365b2328f34ab9b1b73ececbf5cc30&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=c2d8bf7283e14857879d03d1f39c0165 => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:25.328131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b990f7-aa07-4f41-9122-ad1b95527e8e None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:25.356137 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0adfc6-eb85-487c-b926-9ec457e56058 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:25.435415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1f0436-397d-441f-bebb-a0eaa503833d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:25.436042 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1f0436-397d-441f-bebb-a0eaa503833d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:25.436355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1f0436-397d-441f-bebb-a0eaa503833d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:25.436636 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1f0436-397d-441f-bebb-a0eaa503833d None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:25.437355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1f0436-397d-441f-bebb-a0eaa503833d None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:25.437649 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1f0436-397d-441f-bebb-a0eaa503833d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:25.439160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1f0436-397d-441f-bebb-a0eaa503833d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:25.471011 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 343/687] 10.222.0.47 () {56 vars in 1135 bytes} [Sun Aug 21 01:05:25 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 387 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:25.479219 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1f0436-397d-441f-bebb-a0eaa503833d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:25.546823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:25.567185 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b990f7-aa07-4f41-9122-ad1b95527e8e None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:25.631691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccef39c-aab4-46dd-a4f1-0f65d9f6fad0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:25.633435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccef39c-aab4-46dd-a4f1-0f65d9f6fad0 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:25.634806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccef39c-aab4-46dd-a4f1-0f65d9f6fad0 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:25.635871 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccef39c-aab4-46dd-a4f1-0f65d9f6fad0 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:25.637645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccef39c-aab4-46dd-a4f1-0f65d9f6fad0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:25.638397 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccef39c-aab4-46dd-a4f1-0f65d9f6fad0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:25.641473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccef39c-aab4-46dd-a4f1-0f65d9f6fad0 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:25.658754 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1f0436-397d-441f-bebb-a0eaa503833d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:25.663047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 345/688] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:05:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 340 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:25.668919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccef39c-aab4-46dd-a4f1-0f65d9f6fad0 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:25.718328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f2a62a-285f-4d08-a130-b1d0b9925f71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:25.719754 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f2a62a-285f-4d08-a130-b1d0b9925f71 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:25.719872 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f2a62a-285f-4d08-a130-b1d0b9925f71 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:25.720402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f2a62a-285f-4d08-a130-b1d0b9925f71 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:25.740862 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:25.823511 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f2a62a-285f-4d08-a130-b1d0b9925f71 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:25.825504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f2a62a-285f-4d08-a130-b1d0b9925f71 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:25.828836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccef39c-aab4-46dd-a4f1-0f65d9f6fad0 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:25.863561 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:25.884732 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c7e4cd-aa1f-4940-92ca-68473aa44121 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:25.885872 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c7e4cd-aa1f-4940-92ca-68473aa44121 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:25.886835 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c7e4cd-aa1f-4940-92ca-68473aa44121 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:25.887343 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c7e4cd-aa1f-4940-92ca-68473aa44121 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:25.903744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c7e4cd-aa1f-4940-92ca-68473aa44121 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:25.906354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c7e4cd-aa1f-4940-92ca-68473aa44121 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:25.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-80c7e4cd-aa1f-4940-92ca-68473aa44121 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:25.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:25.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:25.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:25.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:25.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:25.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:25.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:25.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:25.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:25.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:25.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:25.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:25.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:25.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:25.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:25.926723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:25.927656 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:25.927656 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:25.927656 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:25.930254 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 346/689] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:25.935665 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c7e4cd-aa1f-4940-92ca-68473aa44121 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:25.958885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 344/690] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 480 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 21 01:05:25.966662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f2a62a-285f-4d08-a130-b1d0b9925f71 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:26.138866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f2a62a-285f-4d08-a130-b1d0b9925f71 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:26.149309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c7e4cd-aa1f-4940-92ca-68473aa44121 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:26.166879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:26.168569 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:26.179861 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d159d7-17fb-4b61-9b5a-8db8e26f7ec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:26.184539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d159d7-17fb-4b61-9b5a-8db8e26f7ec5 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:26.184539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d159d7-17fb-4b61-9b5a-8db8e26f7ec5 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:26.184539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d159d7-17fb-4b61-9b5a-8db8e26f7ec5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:26.192890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d159d7-17fb-4b61-9b5a-8db8e26f7ec5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:26.194496 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d159d7-17fb-4b61-9b5a-8db8e26f7ec5 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:26.199458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74a6bd0-3cb0-43ec-b0eb-3b64f6f2aa12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:26.200370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74a6bd0-3cb0-43ec-b0eb-3b64f6f2aa12 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:26.201039 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74a6bd0-3cb0-43ec-b0eb-3b64f6f2aa12 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:26.201548 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74a6bd0-3cb0-43ec-b0eb-3b64f6f2aa12 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:26.203156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74a6bd0-3cb0-43ec-b0eb-3b64f6f2aa12 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:26.203679 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74a6bd0-3cb0-43ec-b0eb-3b64f6f2aa12 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:26.206129 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74a6bd0-3cb0-43ec-b0eb-3b64f6f2aa12 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:26.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-f5d159d7-17fb-4b61-9b5a-8db8e26f7ec5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:26.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:26.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:26.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:26.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:26.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:26.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:26.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:26.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:26.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:26.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:26.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:26.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:26.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:26.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:26.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:26.212595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:26.213901 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:26.213901 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:26.213901 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:26.214242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 345/691] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:26.215716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723f424e-bd5a-4c82-9c20-e770aa5acd7d None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:26.216095 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723f424e-bd5a-4c82-9c20-e770aa5acd7d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:26.216473 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723f424e-bd5a-4c82-9c20-e770aa5acd7d None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:26.223234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 346/692] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:26.223234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723f424e-bd5a-4c82-9c20-e770aa5acd7d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:26.226101 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 347/693] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:26.232092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3456b884-efe1-4d16-847d-57230c134418 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:26.233893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3456b884-efe1-4d16-847d-57230c134418 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:26.234379 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3456b884-efe1-4d16-847d-57230c134418 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:26.255596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-3456b884-efe1-4d16-847d-57230c134418 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:26.257230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3456b884-efe1-4d16-847d-57230c134418 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:26.345355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 348/694] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:05:26.348030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3456b884-efe1-4d16-847d-57230c134418 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:26.357154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723f424e-bd5a-4c82-9c20-e770aa5acd7d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:26.463794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7225be68-bc53-444a-b489-768b980f7920 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:26.464451 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7225be68-bc53-444a-b489-768b980f7920 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:26.464729 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7225be68-bc53-444a-b489-768b980f7920 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:26.464995 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7225be68-bc53-444a-b489-768b980f7920 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:26.465655 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7225be68-bc53-444a-b489-768b980f7920 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:26.465948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7225be68-bc53-444a-b489-768b980f7920 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:26.467338 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7225be68-bc53-444a-b489-768b980f7920 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:26.475803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:26.491984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3456b884-efe1-4d16-847d-57230c134418 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:26.498534 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 347/695] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:26.503686 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7225be68-bc53-444a-b489-768b980f7920 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:26.563160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b143a8-0160-4e42-b86c-f3760e5e57da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:26.565836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b143a8-0160-4e42-b86c-f3760e5e57da None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:26.566344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b143a8-0160-4e42-b86c-f3760e5e57da None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:26.567382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b143a8-0160-4e42-b86c-f3760e5e57da None admin] PATH_INFO: `/v3/users/nova` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:26.584495 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:26.584908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b143a8-0160-4e42-b86c-f3760e5e57da None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:26.588010 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b143a8-0160-4e42-b86c-f3760e5e57da None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:26.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-62b143a8-0160-4e42-b86c-f3760e5e57da None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:05:26.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:26.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:26.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:26.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:26.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:26.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:26.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:26.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:26.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:26.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:26.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:26.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:26.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:26.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:26.621053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:26.622084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:26.623035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:26.623035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:26.623035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:26.623035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:26.623035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:26.623035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:26.623035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:26.623035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:26.623035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:05:26.623035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:26.624055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 349/696] 10.222.0.47 () {56 vars in 1093 bytes} [Sun Aug 21 01:05:26 2022] GET /identity/v3/users/nova => generated 82 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:26.627932 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b143a8-0160-4e42-b86c-f3760e5e57da None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:26.723709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7225be68-bc53-444a-b489-768b980f7920 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:26.774849 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5147b5-de5b-4a2a-a76a-2c085bb158da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:26.775696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5147b5-de5b-4a2a-a76a-2c085bb158da None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:26.775980 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5147b5-de5b-4a2a-a76a-2c085bb158da None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:26.776289 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5147b5-de5b-4a2a-a76a-2c085bb158da None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:26.784317 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5147b5-de5b-4a2a-a76a-2c085bb158da None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:26.786356 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5147b5-de5b-4a2a-a76a-2c085bb158da None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:26.800064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-8e5147b5-de5b-4a2a-a76a-2c085bb158da None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:26.800064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:26.800064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:26.800064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:26.800064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:26.800064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:26.800064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:26.800064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:26.800064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:26.800064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:26.800064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:26.800064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:26.800064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:26.800064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:26.800064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:26.801558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:26.807015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 348/697] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:26.807989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5147b5-de5b-4a2a-a76a-2c085bb158da None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:26.814528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b143a8-0160-4e42-b86c-f3760e5e57da None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:26.873986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89af1d7-d4b4-4db2-8f4b-89f9d673a3c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:26.874848 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89af1d7-d4b4-4db2-8f4b-89f9d673a3c1 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:26.875338 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89af1d7-d4b4-4db2-8f4b-89f9d673a3c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:26.875667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89af1d7-d4b4-4db2-8f4b-89f9d673a3c1 None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:26.877016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89af1d7-d4b4-4db2-8f4b-89f9d673a3c1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:26.877547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89af1d7-d4b4-4db2-8f4b-89f9d673a3c1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:26.880406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89af1d7-d4b4-4db2-8f4b-89f9d673a3c1 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:26.902576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 350/698] 10.222.0.47 () {56 vars in 1133 bytes} [Sun Aug 21 01:05:26 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:26.903255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89af1d7-d4b4-4db2-8f4b-89f9d673a3c1 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:26.916626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:26.949593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:26.958298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5147b5-de5b-4a2a-a76a-2c085bb158da None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:26.998828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706e8ba4-395b-45f5-86bb-f50e3f70f2d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:27.000078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706e8ba4-395b-45f5-86bb-f50e3f70f2d3 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:27.000545 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706e8ba4-395b-45f5-86bb-f50e3f70f2d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:27.011206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706e8ba4-395b-45f5-86bb-f50e3f70f2d3 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:27.011905 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706e8ba4-395b-45f5-86bb-f50e3f70f2d3 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:27.013404 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706e8ba4-395b-45f5-86bb-f50e3f70f2d3 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-706e8ba4-395b-45f5-86bb-f50e3f70f2d3 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:27.027342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:27.029350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:27.029350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:27.029350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:27.029350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:27.029350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:27.029350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:27.029350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:05:27.029350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:05:27.029350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:05:27.029350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:05:27.029350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:27.029350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:27.029350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:27.029350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:27.029350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:27.029350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:27.030937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:27.030937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:27.030937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:27.030937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:27.030937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:27.030937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:27.030937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:27.030937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:27.030937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:27.030937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:27.030937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:05:27.030937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:27.030937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 21 01:05:27.030937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:27.033923 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89af1d7-d4b4-4db2-8f4b-89f9d673a3c1 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:27.034181 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 349/699] 10.222.0.47 () {56 vars in 1129 bytes} [Sun Aug 21 01:05:26 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:27.034181 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499c697a-801a-4c31-964b-e583a05dfdf4 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:27.034181 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499c697a-801a-4c31-964b-e583a05dfdf4 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:27.034181 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499c697a-801a-4c31-964b-e583a05dfdf4 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:27.034181 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 350/700] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:27.034181 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499c697a-801a-4c31-964b-e583a05dfdf4 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:27.090122 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31419e9-9552-4c0a-958a-472a158cc653 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:27.091247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31419e9-9552-4c0a-958a-472a158cc653 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:27.091800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31419e9-9552-4c0a-958a-472a158cc653 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:27.092817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31419e9-9552-4c0a-958a-472a158cc653 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:27.093955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31419e9-9552-4c0a-958a-472a158cc653 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:27.095279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31419e9-9552-4c0a-958a-472a158cc653 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:27.098205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31419e9-9552-4c0a-958a-472a158cc653 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:27.112644 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 351/701] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:27.114341 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31419e9-9552-4c0a-958a-472a158cc653 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:27.177785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499c697a-801a-4c31-964b-e583a05dfdf4 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:27.207303 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:27.228210 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512ba367-8940-4b70-974e-ba16ac4b9951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:27.228825 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512ba367-8940-4b70-974e-ba16ac4b9951 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:27.229110 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512ba367-8940-4b70-974e-ba16ac4b9951 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:27.229376 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512ba367-8940-4b70-974e-ba16ac4b9951 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:27.249593 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31419e9-9552-4c0a-958a-472a158cc653 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:27.261418 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:27.290154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a01477-4831-4575-bc73-55dcef3505ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:27.290926 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a01477-4831-4575-bc73-55dcef3505ae None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:27.291213 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a01477-4831-4575-bc73-55dcef3505ae None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:27.291794 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a01477-4831-4575-bc73-55dcef3505ae None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:27.292465 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a01477-4831-4575-bc73-55dcef3505ae None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:27.292924 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a01477-4831-4575-bc73-55dcef3505ae None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:27.294446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a01477-4831-4575-bc73-55dcef3505ae None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:27.304189 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 352/702] 10.222.0.47 () {56 vars in 1124 bytes} [Sun Aug 21 01:05:27 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:27.305517 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcc8566-0d1f-496f-89fd-791e164eade9 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:27.305810 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcc8566-0d1f-496f-89fd-791e164eade9 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:27.306085 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcc8566-0d1f-496f-89fd-791e164eade9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:27.307392 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512ba367-8940-4b70-974e-ba16ac4b9951 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:27.308846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512ba367-8940-4b70-974e-ba16ac4b9951 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:27.324965 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-3dcc8566-0d1f-496f-89fd-791e164eade9 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:27.325889 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3dcc8566-0d1f-496f-89fd-791e164eade9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:27.402146 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 353/703] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:27.403664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcc8566-0d1f-496f-89fd-791e164eade9 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:27.448924 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 351/704] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 21 01:05:27.450425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512ba367-8940-4b70-974e-ba16ac4b9951 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:27.501321 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:27.525815 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:27.554833 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcc8566-0d1f-496f-89fd-791e164eade9 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:27.579687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512ba367-8940-4b70-974e-ba16ac4b9951 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:27.603637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f00998-eac0-4ba0-bcd1-f8e6f0584ce9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:27.604309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f00998-eac0-4ba0-bcd1-f8e6f0584ce9 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:27.604584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f00998-eac0-4ba0-bcd1-f8e6f0584ce9 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:27.604853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f00998-eac0-4ba0-bcd1-f8e6f0584ce9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:27.605588 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f00998-eac0-4ba0-bcd1-f8e6f0584ce9 None admin] RBAC: Adding query filter params (role.id=c2d8bf7283e14857879d03d1f39c0165, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=06a5972a55824711b06670b925322739) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:27.605890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f00998-eac0-4ba0-bcd1-f8e6f0584ce9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:27.608272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f00998-eac0-4ba0-bcd1-f8e6f0584ce9 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:27.624325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 354/705] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:27 2022] GET /identity/v3/role_assignments?user.id=06a5972a55824711b06670b925322739&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=c2d8bf7283e14857879d03d1f39c0165 => generated 608 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:27.624325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f00998-eac0-4ba0-bcd1-f8e6f0584ce9 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:27.649551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dd2674-aefb-4b13-95b5-7c22c34b2154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:27.650575 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dd2674-aefb-4b13-95b5-7c22c34b2154 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:27.651054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dd2674-aefb-4b13-95b5-7c22c34b2154 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:27.651523 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dd2674-aefb-4b13-95b5-7c22c34b2154 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:27.652477 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dd2674-aefb-4b13-95b5-7c22c34b2154 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:27.653460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dd2674-aefb-4b13-95b5-7c22c34b2154 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:27.655389 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dd2674-aefb-4b13-95b5-7c22c34b2154 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:27.670802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 352/706] 10.222.0.47 () {56 vars in 1124 bytes} [Sun Aug 21 01:05:27 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:27.699881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dd2674-aefb-4b13-95b5-7c22c34b2154 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:27.763090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f00998-eac0-4ba0-bcd1-f8e6f0584ce9 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:27.804515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:27.821193 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc57b96d-8843-4253-b144-959de0dfa9d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:27.821931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc57b96d-8843-4253-b144-959de0dfa9d3 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:27.822216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc57b96d-8843-4253-b144-959de0dfa9d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:27.822817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc57b96d-8843-4253-b144-959de0dfa9d3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:27.838254 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc57b96d-8843-4253-b144-959de0dfa9d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:27.839751 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc57b96d-8843-4253-b144-959de0dfa9d3 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:27.865668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-cc57b96d-8843-4253-b144-959de0dfa9d3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:27.865668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:27.865668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:27.865668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:27.865668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:27.865668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:27.865668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:27.865668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:27.865668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:27.865668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:27.865668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:27.865668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:27.865668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:27.865668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:27.865668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:27.865668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:27.866940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:27.867850 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:27.867850 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:27.867850 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:27.868034 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dd2674-aefb-4b13-95b5-7c22c34b2154 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:27.868174 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 355/707] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:27.873906 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc57b96d-8843-4253-b144-959de0dfa9d3 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:27.910056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:27.924345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070e28be-bbf2-4cca-941b-37eeb2f847ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:27.927364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070e28be-bbf2-4cca-941b-37eeb2f847ce None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:27.927364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070e28be-bbf2-4cca-941b-37eeb2f847ce None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:27.927364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070e28be-bbf2-4cca-941b-37eeb2f847ce None admin] PATH_INFO: `/v3/services/network` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:27.927364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070e28be-bbf2-4cca-941b-37eeb2f847ce None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:27.929718 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070e28be-bbf2-4cca-941b-37eeb2f847ce None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-070e28be-bbf2-4cca-941b-37eeb2f847ce None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:27.941355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:27.942741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:27.942741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:27.942741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:27.942741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:27.942741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:27.942741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:27.942741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:05:27.942741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:05:27.942741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:05:27.942741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:05:27.942741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:27.942741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:27.942741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:27.942741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:27.942741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:27.942741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:27.943683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:27.943683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:27.943683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:27.943683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:27.943683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:27.943683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:27.943683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:27.943683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:27.943683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:27.943683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:27.943683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:05:27.943683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:27.943683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 21 01:05:27.943683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:27.943683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 353/708] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:27 2022] GET /identity/v3/services/network => generated 88 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:27.951374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070e28be-bbf2-4cca-941b-37eeb2f847ce None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:28.025890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc57b96d-8843-4253-b144-959de0dfa9d3 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:28.076470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec20cc3b-bd39-4f61-9fa5-21d5e8ac8dd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:28.077206 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec20cc3b-bd39-4f61-9fa5-21d5e8ac8dd3 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:28.077985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec20cc3b-bd39-4f61-9fa5-21d5e8ac8dd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:28.078353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec20cc3b-bd39-4f61-9fa5-21d5e8ac8dd3 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:28.079086 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec20cc3b-bd39-4f61-9fa5-21d5e8ac8dd3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:28.079616 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec20cc3b-bd39-4f61-9fa5-21d5e8ac8dd3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:28.082122 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec20cc3b-bd39-4f61-9fa5-21d5e8ac8dd3 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:28.089277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070e28be-bbf2-4cca-941b-37eeb2f847ce None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:28.104191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 356/709] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:28.112155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec20cc3b-bd39-4f61-9fa5-21d5e8ac8dd3 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:28.155965 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:28.174829 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d871a57-bc57-4740-a6fb-c9976d7e9352 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:28.175889 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d871a57-bc57-4740-a6fb-c9976d7e9352 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:28.176400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d871a57-bc57-4740-a6fb-c9976d7e9352 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:28.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d871a57-bc57-4740-a6fb-c9976d7e9352 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:28.177701 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d871a57-bc57-4740-a6fb-c9976d7e9352 None admin] RBAC: Adding query filter params (name=network) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:28.178233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d871a57-bc57-4740-a6fb-c9976d7e9352 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:28.179906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d871a57-bc57-4740-a6fb-c9976d7e9352 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:28.193721 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 354/710] 10.222.0.47 () {56 vars in 1112 bytes} [Sun Aug 21 01:05:27 2022] GET /identity/v3/services?name=network => generated 124 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:28.195121 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60c14b8-d213-46d0-99f0-176ec13345cf None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:28.195404 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60c14b8-d213-46d0-99f0-176ec13345cf None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:28.195673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60c14b8-d213-46d0-99f0-176ec13345cf None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:28.201336 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 355/711] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:28.202226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60c14b8-d213-46d0-99f0-176ec13345cf None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:28.234600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec20cc3b-bd39-4f61-9fa5-21d5e8ac8dd3 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:28.268431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:28.295051 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cd3e8b-2d1b-4eb6-8e26-827d2185e1ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:28.295614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cd3e8b-2d1b-4eb6-8e26-827d2185e1ca None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:28.295894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cd3e8b-2d1b-4eb6-8e26-827d2185e1ca None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:28.296192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cd3e8b-2d1b-4eb6-8e26-827d2185e1ca None admin] PATH_INFO: `/v3/projects/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:28.305463 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cd3e8b-2d1b-4eb6-8e26-827d2185e1ca None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:28.307310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cd3e8b-2d1b-4eb6-8e26-827d2185e1ca None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:28.325266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-a7cd3e8b-2d1b-4eb6-8e26-827d2185e1ca None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:28.325266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:28.325266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:28.325266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:28.325266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:28.325266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:28.325266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:28.325266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:28.325266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:28.325266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:28.325266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:28.325266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:28.325266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:28.325266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:28.325266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:28.326507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:28.327647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:28.327647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:28.327647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:28.327647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:28.327647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:28.327647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:28.327647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:28.327647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:28.328429 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 357/712] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:28.331914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f3a1be-414d-4dcd-a0e7-c5eb1a5398ed None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:28.331914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f3a1be-414d-4dcd-a0e7-c5eb1a5398ed None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:28.331914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f3a1be-414d-4dcd-a0e7-c5eb1a5398ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:28.347654 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60c14b8-d213-46d0-99f0-176ec13345cf None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:28.356349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-35f3a1be-414d-4dcd-a0e7-c5eb1a5398ed None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:28.358108 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-35f3a1be-414d-4dcd-a0e7-c5eb1a5398ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:28.386150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120935f4-de4a-4f66-833b-587deb263e04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:28.387073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120935f4-de4a-4f66-833b-587deb263e04 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:28.387578 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120935f4-de4a-4f66-833b-587deb263e04 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:28.388078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120935f4-de4a-4f66-833b-587deb263e04 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:28.388904 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120935f4-de4a-4f66-833b-587deb263e04 None admin] RBAC: Adding query filter params (type=network) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:28.389378 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120935f4-de4a-4f66-833b-587deb263e04 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:28.391154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120935f4-de4a-4f66-833b-587deb263e04 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:28.396813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 356/713] 10.222.0.47 () {56 vars in 1112 bytes} [Sun Aug 21 01:05:28 2022] GET /identity/v3/services?type=network => generated 124 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:28.398029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120935f4-de4a-4f66-833b-587deb263e04 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:28.452138 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 358/714] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:28.561929 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120935f4-de4a-4f66-833b-587deb263e04 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:28.588390 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:28.612408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e7daba-3c75-439f-b376-ed567f82d5fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:28.613078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e7daba-3c75-439f-b376-ed567f82d5fd None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:28.613350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e7daba-3c75-439f-b376-ed567f82d5fd None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:28.613617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e7daba-3c75-439f-b376-ed567f82d5fd None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:28.614305 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e7daba-3c75-439f-b376-ed567f82d5fd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:28.614601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e7daba-3c75-439f-b376-ed567f82d5fd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:28.616050 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e7daba-3c75-439f-b376-ed567f82d5fd None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:28.632982 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 357/715] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:28.638520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f3a1be-414d-4dcd-a0e7-c5eb1a5398ed None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:28.673113 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:28.782452 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e7daba-3c75-439f-b376-ed567f82d5fd None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:28.783296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f3a1be-414d-4dcd-a0e7-c5eb1a5398ed None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:28.824733 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78e47a0-5082-4975-b9c8-e0728e23092d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:28.825533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78e47a0-5082-4975-b9c8-e0728e23092d None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:28.825822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78e47a0-5082-4975-b9c8-e0728e23092d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:28.826090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78e47a0-5082-4975-b9c8-e0728e23092d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:28.826821 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78e47a0-5082-4975-b9c8-e0728e23092d None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=fabdcdc25d1d4ef6bc66bd6cadecaaf8) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:28.827144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78e47a0-5082-4975-b9c8-e0728e23092d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:28.829416 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78e47a0-5082-4975-b9c8-e0728e23092d None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:28.836437 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 359/716] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:28 2022] GET /identity/v3/role_assignments?user.id=fabdcdc25d1d4ef6bc66bd6cadecaaf8&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 259 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:28.980504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e7daba-3c75-439f-b376-ed567f82d5fd None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:29.029736 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf40ca4-5ded-4a21-a7ea-e89438753a4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:29.030852 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf40ca4-5ded-4a21-a7ea-e89438753a4a None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:29.031346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf40ca4-5ded-4a21-a7ea-e89438753a4a None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:29.031816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf40ca4-5ded-4a21-a7ea-e89438753a4a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:29.048756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf40ca4-5ded-4a21-a7ea-e89438753a4a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:29.050638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf40ca4-5ded-4a21-a7ea-e89438753a4a None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:29.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-adf40ca4-5ded-4a21-a7ea-e89438753a4a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:29.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:29.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:29.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:29.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:29.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:29.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:29.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:29.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:29.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:29.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:29.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:29.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:29.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:29.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:29.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:29.068601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:29.069518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:29.069518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:29.069518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:29.074934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 358/717] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:29.078667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78e47a0-5082-4975-b9c8-e0728e23092d None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:29.211094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:29.224649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78e47a0-5082-4975-b9c8-e0728e23092d None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:29.275661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1eeecc-5be7-47e5-ab41-1d6d0f530dc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:29.276327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1eeecc-5be7-47e5-ab41-1d6d0f530dc4 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:29.276615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1eeecc-5be7-47e5-ab41-1d6d0f530dc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:29.276896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1eeecc-5be7-47e5-ab41-1d6d0f530dc4 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:29.277529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1eeecc-5be7-47e5-ab41-1d6d0f530dc4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:29.277826 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1eeecc-5be7-47e5-ab41-1d6d0f530dc4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:29.279201 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1eeecc-5be7-47e5-ab41-1d6d0f530dc4 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:29.292114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 360/718] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:29.298167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf40ca4-5ded-4a21-a7ea-e89438753a4a None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:29.364835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:29.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf40ca4-5ded-4a21-a7ea-e89438753a4a None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:29.581708 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3002ee4-98e3-4eb6-95d2-0823d79a7233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:29.582710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3002ee4-98e3-4eb6-95d2-0823d79a7233 None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:29.583140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3002ee4-98e3-4eb6-95d2-0823d79a7233 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:29.583624 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3002ee4-98e3-4eb6-95d2-0823d79a7233 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:29.584743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3002ee4-98e3-4eb6-95d2-0823d79a7233 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:29.587346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3002ee4-98e3-4eb6-95d2-0823d79a7233 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:29.693524 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 359/719] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:05:29 2022] POST /identity/v3/users => generated 254 bytes in 396 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:05:30.225077 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc279f8-971a-4c17-8c2c-f482e310260c None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:30.225596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc279f8-971a-4c17-8c2c-f482e310260c None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:30.225596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc279f8-971a-4c17-8c2c-f482e310260c None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:30.227297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 361/720] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:30.232804 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61f9c39-3a60-47ab-8372-18c370b86d68 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:30.233098 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61f9c39-3a60-47ab-8372-18c370b86d68 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:30.233357 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61f9c39-3a60-47ab-8372-18c370b86d68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:30.257947 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f61f9c39-3a60-47ab-8372-18c370b86d68 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:30.259162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f61f9c39-3a60-47ab-8372-18c370b86d68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:30.355283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 360/721] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:30.418443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a861502c-3060-40c0-a5e3-1237f24c9cea None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:30.418720 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a861502c-3060-40c0-a5e3-1237f24c9cea None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:30.418996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a861502c-3060-40c0-a5e3-1237f24c9cea None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:30.420691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 362/722] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:30.426682 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071dc213-4471-43cb-8d8c-a35209e4bb3e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:30.426973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071dc213-4471-43cb-8d8c-a35209e4bb3e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:30.427251 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071dc213-4471-43cb-8d8c-a35209e4bb3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:30.449433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-071dc213-4471-43cb-8d8c-a35209e4bb3e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:30.450645 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-071dc213-4471-43cb-8d8c-a35209e4bb3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:30.464671 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:30.542636 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 361/723] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:30.713691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a861502c-3060-40c0-a5e3-1237f24c9cea None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:30.865870 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a861502c-3060-40c0-a5e3-1237f24c9cea None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:30.876911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071dc213-4471-43cb-8d8c-a35209e4bb3e None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:30.913825 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522de227-cf15-4b55-acdd-5c2946a99087 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:30.914453 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522de227-cf15-4b55-acdd-5c2946a99087 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:30.914736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522de227-cf15-4b55-acdd-5c2946a99087 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:30.915009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522de227-cf15-4b55-acdd-5c2946a99087 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:30.915673 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522de227-cf15-4b55-acdd-5c2946a99087 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:30.917094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522de227-cf15-4b55-acdd-5c2946a99087 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:30.996694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071dc213-4471-43cb-8d8c-a35209e4bb3e None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:31.007990 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 363/724] 10.222.0.47 () {60 vars in 1141 bytes} [Sun Aug 21 01:05:30 2022] POST /identity/v3/services => generated 251 bytes in 295 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:05:31.055575 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ff5f8e-7e46-4bf3-8818-0375631ed79b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:31.056577 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ff5f8e-7e46-4bf3-8818-0375631ed79b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:31.057065 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ff5f8e-7e46-4bf3-8818-0375631ed79b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:31.057518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ff5f8e-7e46-4bf3-8818-0375631ed79b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:31.070446 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ff5f8e-7e46-4bf3-8818-0375631ed79b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:31.080156 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:31.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-e9ff5f8e-7e46-4bf3-8818-0375631ed79b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:05:31.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:31.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:31.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:31.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:31.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:31.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:31.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:31.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:31.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:31.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:31.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:31.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:31.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:31.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:31.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:31.083814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:31.083814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:31.083814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:31.083814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:31.083814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:31.083814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:31.083814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:31.083814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:31.083814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:31.083814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:31.083814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:05:31.083814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:31.086266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 362/725] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:05:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:31.092377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522de227-cf15-4b55-acdd-5c2946a99087 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:31.118926 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:31.216534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522de227-cf15-4b55-acdd-5c2946a99087 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:31.252890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456cb1ad-f959-428b-80e6-95b39ac53d96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:31.254193 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456cb1ad-f959-428b-80e6-95b39ac53d96 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:31.254612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456cb1ad-f959-428b-80e6-95b39ac53d96 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:31.255016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456cb1ad-f959-428b-80e6-95b39ac53d96 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:31.255810 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456cb1ad-f959-428b-80e6-95b39ac53d96 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:31.256296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456cb1ad-f959-428b-80e6-95b39ac53d96 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:31.257806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456cb1ad-f959-428b-80e6-95b39ac53d96 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:31.267053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 364/726] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:05:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:31.273562 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ff5f8e-7e46-4bf3-8818-0375631ed79b None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:31.385937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ff5f8e-7e46-4bf3-8818-0375631ed79b None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:31.427318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fd4eb6-218e-45df-9a85-4fbea929fac6 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:31.427618 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fd4eb6-218e-45df-9a85-4fbea929fac6 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:31.427857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fd4eb6-218e-45df-9a85-4fbea929fac6 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:31.429212 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 365/727] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:31.434427 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a14ed0c-6be1-485f-be7d-c62dd3a7cba0 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:31.434717 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a14ed0c-6be1-485f-be7d-c62dd3a7cba0 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:31.434992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a14ed0c-6be1-485f-be7d-c62dd3a7cba0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:31.459143 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca70b501-a98e-4fe1-923b-b45325b3525b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:31.459783 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca70b501-a98e-4fe1-923b-b45325b3525b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:31.460059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca70b501-a98e-4fe1-923b-b45325b3525b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:31.460350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca70b501-a98e-4fe1-923b-b45325b3525b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:31.463134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-0a14ed0c-6be1-485f-be7d-c62dd3a7cba0 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:31.464369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-0a14ed0c-6be1-485f-be7d-c62dd3a7cba0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:31.480095 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca70b501-a98e-4fe1-923b-b45325b3525b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:31.481666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca70b501-a98e-4fe1-923b-b45325b3525b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:31.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-ca70b501-a98e-4fe1-923b-b45325b3525b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:31.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:31.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:31.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:31.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:31.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:31.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:31.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:31.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:31.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:31.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:31.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:31.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:31.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:31.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:31.509778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:31.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:31.511712 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:31.511712 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:31.511712 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:31.516131 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 363/728] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:31.518117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca70b501-a98e-4fe1-923b-b45325b3525b None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:31.578558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:31.589293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 366/729] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:31 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 21 01:05:31.639191 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca70b501-a98e-4fe1-923b-b45325b3525b None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:31.674217 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13818a1f-eaf9-473f-9d73-969b811e8865 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:31.675006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13818a1f-eaf9-473f-9d73-969b811e8865 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:31.675514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13818a1f-eaf9-473f-9d73-969b811e8865 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:31.675907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13818a1f-eaf9-473f-9d73-969b811e8865 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:31.677026 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13818a1f-eaf9-473f-9d73-969b811e8865 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:31.677379 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13818a1f-eaf9-473f-9d73-969b811e8865 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:31.678738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13818a1f-eaf9-473f-9d73-969b811e8865 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:31.692399 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 364/730] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:31.700739 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a14ed0c-6be1-485f-be7d-c62dd3a7cba0 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:31.707948 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:31.829262 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a14ed0c-6be1-485f-be7d-c62dd3a7cba0 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:31.852294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722c4dcc-6984-4fb0-94e9-8ed55438e476 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:31.852614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722c4dcc-6984-4fb0-94e9-8ed55438e476 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:31.852827 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722c4dcc-6984-4fb0-94e9-8ed55438e476 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:31.854541 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 365/731] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:31.859074 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cd80ab-6b6a-4ac9-876d-818db06a80f9 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:31.859346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cd80ab-6b6a-4ac9-876d-818db06a80f9 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:31.859616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cd80ab-6b6a-4ac9-876d-818db06a80f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:31.872525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685fb1c6-2a08-4e05-8928-44585d79967c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:31.873158 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685fb1c6-2a08-4e05-8928-44585d79967c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:31.873432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685fb1c6-2a08-4e05-8928-44585d79967c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:31.873696 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685fb1c6-2a08-4e05-8928-44585d79967c None admin] PATH_INFO: `/v3/users/swift` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:31.887700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685fb1c6-2a08-4e05-8928-44585d79967c None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:31.889298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685fb1c6-2a08-4e05-8928-44585d79967c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:31.891879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-38cd80ab-6b6a-4ac9-876d-818db06a80f9 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:31.893096 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-38cd80ab-6b6a-4ac9-876d-818db06a80f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:31.907541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-685fb1c6-2a08-4e05-8928-44585d79967c None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 21 01:05:31.907541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:31.907541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:31.907541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:31.907541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:31.907541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:31.907541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:31.907541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:31.907541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:31.907541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:31.907541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:31.907541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:31.907541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:31.907541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:31.907541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:31.907541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:31.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:31.912203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:31.912203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:31.912203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:31.912203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:31.912203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:31.912203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:31.912203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:31.912203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:31.912203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 21 01:05:31.912203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:31.917314 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 367/732] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:05:31 2022] GET /identity/v3/users/swift => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:31.918612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685fb1c6-2a08-4e05-8928-44585d79967c None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:31.964808 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 366/733] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:31 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 21 01:05:31.965721 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cd80ab-6b6a-4ac9-876d-818db06a80f9 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:31.993418 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:32.042056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:32.077007 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685fb1c6-2a08-4e05-8928-44585d79967c None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:32.156597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f06f29-6696-4dc8-837c-076c90c19232 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:32.157022 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f06f29-6696-4dc8-837c-076c90c19232 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:32.157300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f06f29-6696-4dc8-837c-076c90c19232 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:32.157562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f06f29-6696-4dc8-837c-076c90c19232 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:32.158223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f06f29-6696-4dc8-837c-076c90c19232 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:32.159632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f06f29-6696-4dc8-837c-076c90c19232 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:32.168912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cd80ab-6b6a-4ac9-876d-818db06a80f9 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:32.208463 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f1e407-859a-4f20-9efc-cf15a15001de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:32.209114 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f1e407-859a-4f20-9efc-cf15a15001de None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:32.209527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f1e407-859a-4f20-9efc-cf15a15001de None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:32.209921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f1e407-859a-4f20-9efc-cf15a15001de None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:32.210726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f1e407-859a-4f20-9efc-cf15a15001de None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:32.211135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f1e407-859a-4f20-9efc-cf15a15001de None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:32.212761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f1e407-859a-4f20-9efc-cf15a15001de None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:32.246778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 367/734] 10.222.0.47 () {56 vars in 1135 bytes} [Sun Aug 21 01:05:31 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:32.247772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f1e407-859a-4f20-9efc-cf15a15001de None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:32.256522 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 368/735] 10.222.0.47 () {60 vars in 1141 bytes} [Sun Aug 21 01:05:31 2022] POST /identity/v3/services => generated 240 bytes in 333 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:05:32.257682 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f06f29-6696-4dc8-837c-076c90c19232 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:32.365290 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:32.368786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f1e407-859a-4f20-9efc-cf15a15001de None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:32.403875 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f06f29-6696-4dc8-837c-076c90c19232 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:32.411216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a136aa-850b-4633-aa56-2732a5426cba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:32.411793 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a136aa-850b-4633-aa56-2732a5426cba None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:32.412065 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a136aa-850b-4633-aa56-2732a5426cba None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:32.412383 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a136aa-850b-4633-aa56-2732a5426cba None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:32.426607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a136aa-850b-4633-aa56-2732a5426cba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:32.430206 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:32.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-f2a136aa-850b-4633-aa56-2732a5426cba None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:05:32.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:32.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:32.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:32.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:32.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:32.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:32.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:32.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:32.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:32.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:32.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:32.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:32.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:32.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:32.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:32.433309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:32.433309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:32.433309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:32.433309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:32.433309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:32.433309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:32.433309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:32.433309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:32.433309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:32.433309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:32.433309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:32.433309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:32.433309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:32.433309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:32.433309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:32.433309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:32.434150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:32.434150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:32.434150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:32.434150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:32.434150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:32.434150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:32.434150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:32.434150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:32.434150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:32.434150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:32.434150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:05:32.434150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:32.434150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 368/736] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:05:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:32.444591 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b44a3d-3e04-4796-ad7a-8f2394a93434 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:32.444896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b44a3d-3e04-4796-ad7a-8f2394a93434 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:32.445525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b44a3d-3e04-4796-ad7a-8f2394a93434 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:32.447880 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 369/737] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:05:32.448661 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b44a3d-3e04-4796-ad7a-8f2394a93434 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:32.458277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf30b67-d7e6-4e20-887b-d43edb668786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:32.459251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf30b67-d7e6-4e20-887b-d43edb668786 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:32.459707 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf30b67-d7e6-4e20-887b-d43edb668786 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:32.460181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf30b67-d7e6-4e20-887b-d43edb668786 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:32.559274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf30b67-d7e6-4e20-887b-d43edb668786 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:32.561265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf30b67-d7e6-4e20-887b-d43edb668786 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:32.583522 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b44a3d-3e04-4796-ad7a-8f2394a93434 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:32.623740 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6307095b-ef8d-46ff-b325-d2b32911c75a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:32.624874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6307095b-ef8d-46ff-b325-d2b32911c75a None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:32.625355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6307095b-ef8d-46ff-b325-d2b32911c75a None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:32.625848 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6307095b-ef8d-46ff-b325-d2b32911c75a None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:32.626722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6307095b-ef8d-46ff-b325-d2b32911c75a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:32.627221 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6307095b-ef8d-46ff-b325-d2b32911c75a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:32.629418 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6307095b-ef8d-46ff-b325-d2b32911c75a None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:32.639200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 370/738] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:05:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:32.640911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f85058-c776-402c-8930-201f1f1d64ba None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:32.641462 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f85058-c776-402c-8930-201f1f1d64ba None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:32.641940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f85058-c776-402c-8930-201f1f1d64ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:32.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-27f85058-c776-402c-8930-201f1f1d64ba None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:32.664069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-27f85058-c776-402c-8930-201f1f1d64ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:32.684489 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:32.689673 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 369/739] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:32 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:05:32.690913 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf30b67-d7e6-4e20-887b-d43edb668786 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:32.715826 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:32.748561 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 371/740] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:32 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:32.749442 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f85058-c776-402c-8930-201f1f1d64ba None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:32.843907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf30b67-d7e6-4e20-887b-d43edb668786 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:32.862790 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f85058-c776-402c-8930-201f1f1d64ba None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:32.903249 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f2e760-1e65-4f47-bd0c-142b804cd36d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:32.903912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f2e760-1e65-4f47-bd0c-142b804cd36d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:32.904190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f2e760-1e65-4f47-bd0c-142b804cd36d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:32.904469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f2e760-1e65-4f47-bd0c-142b804cd36d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:32.915436 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f2e760-1e65-4f47-bd0c-142b804cd36d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:32.916847 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f2e760-1e65-4f47-bd0c-142b804cd36d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:32.917376 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a36376-2ab5-4ec8-953d-6cdbb2dc4942 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:32.918027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a36376-2ab5-4ec8-953d-6cdbb2dc4942 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:32.918301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a36376-2ab5-4ec8-953d-6cdbb2dc4942 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:32.918568 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a36376-2ab5-4ec8-953d-6cdbb2dc4942 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:32.933340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-65f2e760-1e65-4f47-bd0c-142b804cd36d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:32.933340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:32.933340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:32.933340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:32.933340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:32.933340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:32.933340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:32.933340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:32.933340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:32.933340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:32.933340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:32.933340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:32.933340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:32.933340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:32.933340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:32.933340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:32.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:32.935566 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:32.935566 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:32.935566 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:32.935566 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 372/741] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:32.941637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f2e760-1e65-4f47-bd0c-142b804cd36d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:32.944007 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a36376-2ab5-4ec8-953d-6cdbb2dc4942 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:32.945497 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a36376-2ab5-4ec8-953d-6cdbb2dc4942 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:32.946100 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:32.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-86a36376-2ab5-4ec8-953d-6cdbb2dc4942 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:32.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:32.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:32.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:32.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:32.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:32.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:32.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:32.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:32.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:32.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:32.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:32.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:32.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:32.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:32.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:32.981047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:32.981983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:32.981983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:32.981983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:32.987703 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 370/742] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:32.988699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a36376-2ab5-4ec8-953d-6cdbb2dc4942 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:33.053373 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f2e760-1e65-4f47-bd0c-142b804cd36d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:33.133244 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:33.133578 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a30b61d-b210-444b-94f3-8a1d13cc480c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:33.134172 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a30b61d-b210-444b-94f3-8a1d13cc480c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:33.134431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a30b61d-b210-444b-94f3-8a1d13cc480c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:33.134698 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a30b61d-b210-444b-94f3-8a1d13cc480c None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:33.135354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a30b61d-b210-444b-94f3-8a1d13cc480c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:33.135650 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a30b61d-b210-444b-94f3-8a1d13cc480c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:33.137043 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a30b61d-b210-444b-94f3-8a1d13cc480c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:33.166609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 373/743] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:33.168087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a30b61d-b210-444b-94f3-8a1d13cc480c None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:33.178301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a36376-2ab5-4ec8-953d-6cdbb2dc4942 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:33.219693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7113a126-97a2-492e-9b7a-d330b6cca798 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:33.221407 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7113a126-97a2-492e-9b7a-d330b6cca798 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:33.221407 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7113a126-97a2-492e-9b7a-d330b6cca798 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:33.221407 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7113a126-97a2-492e-9b7a-d330b6cca798 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:33.222305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7113a126-97a2-492e-9b7a-d330b6cca798 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:33.222443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7113a126-97a2-492e-9b7a-d330b6cca798 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:33.223693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7113a126-97a2-492e-9b7a-d330b6cca798 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:33.236902 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 371/744] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:33.238441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7113a126-97a2-492e-9b7a-d330b6cca798 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:33.259407 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:33.306425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a30b61d-b210-444b-94f3-8a1d13cc480c None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:33.345089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7113a126-97a2-492e-9b7a-d330b6cca798 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:33.348551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5193f4-16fe-4dd5-95f5-a375d642c01c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:33.349561 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5193f4-16fe-4dd5-95f5-a375d642c01c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:33.350024 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5193f4-16fe-4dd5-95f5-a375d642c01c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:33.350493 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5193f4-16fe-4dd5-95f5-a375d642c01c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:33.359413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5193f4-16fe-4dd5-95f5-a375d642c01c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:33.361411 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5193f4-16fe-4dd5-95f5-a375d642c01c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:33.377323 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:33.377788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-eb5193f4-16fe-4dd5-95f5-a375d642c01c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:33.377788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:33.377788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:33.377788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:33.377788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:33.377788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:33.377788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:33.377788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:33.377788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:33.377788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:33.377788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:33.377788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:33.377788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:33.377788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:33.377788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:33.379133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:33.380428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:33.380428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:33.380428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:33.380428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:33.380428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:33.380428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:33.380428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:33.380428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:33.388478 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 374/745] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:33.388478 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5193f4-16fe-4dd5-95f5-a375d642c01c None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:33.391625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6584d26-d4ad-4ab7-a5d8-f149517b0b9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:33.391625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6584d26-d4ad-4ab7-a5d8-f149517b0b9d None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:33.391625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6584d26-d4ad-4ab7-a5d8-f149517b0b9d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:33.391625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6584d26-d4ad-4ab7-a5d8-f149517b0b9d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:33.403157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6584d26-d4ad-4ab7-a5d8-f149517b0b9d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:33.408220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-e6584d26-d4ad-4ab7-a5d8-f149517b0b9d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:33.408220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:33.408220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:33.408220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:33.408220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:33.408220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:33.408220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:33.408220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:33.408220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:33.408220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:33.408220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:33.408220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:33.408220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:33.408220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:33.408220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:33.408220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:33.409215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:33.409215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:33.409215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:33.409215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:33.409215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:33.409215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:33.409215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:33.409215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:33.409215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:33.409215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:33.409215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:33.409215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:33.409215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:33.409215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:33.409215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:33.409215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:33.410093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:33.410093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:33.410093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:33.410093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:33.410093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:33.410093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:33.410093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:33.410093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:33.410093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:33.410093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:33.410093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:33.410093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:33.412354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 372/746] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:33.415871 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6584d26-d4ad-4ab7-a5d8-f149517b0b9d None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:33.511604 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5193f4-16fe-4dd5-95f5-a375d642c01c None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:33.524279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6584d26-d4ad-4ab7-a5d8-f149517b0b9d None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:33.540063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:33.552965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301b0e1d-96be-46f8-8559-b5e6e673fd93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:33.553934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301b0e1d-96be-46f8-8559-b5e6e673fd93 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:33.557381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301b0e1d-96be-46f8-8559-b5e6e673fd93 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:33.557381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301b0e1d-96be-46f8-8559-b5e6e673fd93 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:33.561262 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f29e46-5f6f-4c5d-849a-6ed1abe340c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:33.562000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f29e46-5f6f-4c5d-849a-6ed1abe340c6 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:33.562391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f29e46-5f6f-4c5d-849a-6ed1abe340c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:33.562752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f29e46-5f6f-4c5d-849a-6ed1abe340c6 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:33.563667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f29e46-5f6f-4c5d-849a-6ed1abe340c6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:33.564131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f29e46-5f6f-4c5d-849a-6ed1abe340c6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:33.565364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301b0e1d-96be-46f8-8559-b5e6e673fd93 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:33.565806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f29e46-5f6f-4c5d-849a-6ed1abe340c6 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:33.567864 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301b0e1d-96be-46f8-8559-b5e6e673fd93 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:33.584141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 373/747] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:33.584141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f29e46-5f6f-4c5d-849a-6ed1abe340c6 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:33.585942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-301b0e1d-96be-46f8-8559-b5e6e673fd93 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:05:33.585942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:33.585942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:33.585942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:33.585942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:33.585942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:33.585942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:33.585942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:33.585942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:33.585942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:33.585942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:33.585942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:33.585942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:33.585942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:33.585942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:33.585942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:33.586881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:33.587833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:33.587833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:33.587833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:33.587833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:33.587833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:33.587833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:33.587833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:33.587833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:33.587833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:05:33.587833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:33.591499 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 375/748] 10.222.0.47 () {56 vars in 1093 bytes} [Sun Aug 21 01:05:33 2022] GET /identity/v3/users/nova => generated 82 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:33.596313 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301b0e1d-96be-46f8-8559-b5e6e673fd93 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:33.648297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:33.709145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301b0e1d-96be-46f8-8559-b5e6e673fd93 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:33.709972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f29e46-5f6f-4c5d-849a-6ed1abe340c6 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:33.748012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb52220-88f3-4f55-b35c-a0259b574e47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:33.749166 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb52220-88f3-4f55-b35c-a0259b574e47 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:33.749630 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb52220-88f3-4f55-b35c-a0259b574e47 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:33.750105 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb52220-88f3-4f55-b35c-a0259b574e47 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:33.751047 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb52220-88f3-4f55-b35c-a0259b574e47 None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=06a5972a55824711b06670b925322739) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:33.751355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb52220-88f3-4f55-b35c-a0259b574e47 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:33.752845 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb52220-88f3-4f55-b35c-a0259b574e47 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:33.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 376/749] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:33 2022] GET /identity/v3/role_assignments?user.id=06a5972a55824711b06670b925322739&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 259 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:33.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6aa7dc7-71a6-4996-837b-1b74fe6f5c83 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:33.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6aa7dc7-71a6-4996-837b-1b74fe6f5c83 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:33.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6aa7dc7-71a6-4996-837b-1b74fe6f5c83 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:33.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 377/750] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:33.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6aa7dc7-71a6-4996-837b-1b74fe6f5c83 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:33.771334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e72584-52e4-42f0-a2b3-b062b1c8993a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:33.771978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e72584-52e4-42f0-a2b3-b062b1c8993a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:33.772292 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e72584-52e4-42f0-a2b3-b062b1c8993a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:33.772807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e72584-52e4-42f0-a2b3-b062b1c8993a None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:33.773242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e72584-52e4-42f0-a2b3-b062b1c8993a None admin] RBAC: Adding query filter params (name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:33.773539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e72584-52e4-42f0-a2b3-b062b1c8993a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:33.775102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e72584-52e4-42f0-a2b3-b062b1c8993a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:33.790257 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 374/751] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:33.791991 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfc395e-1bc6-43d5-ab5f-badfea50bb55 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:33.792306 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfc395e-1bc6-43d5-ab5f-badfea50bb55 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:33.792598 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfc395e-1bc6-43d5-ab5f-badfea50bb55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:33.813542 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-6dfc395e-1bc6-43d5-ab5f-badfea50bb55 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:33.814806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6dfc395e-1bc6-43d5-ab5f-badfea50bb55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:33.840946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:33.912369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 375/752] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:33 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:33.913431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfc395e-1bc6-43d5-ab5f-badfea50bb55 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:33.920220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6aa7dc7-71a6-4996-837b-1b74fe6f5c83 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:33.980581 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658dd1e7-f482-4a5e-82fc-84a257b3fa17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:33.981860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658dd1e7-f482-4a5e-82fc-84a257b3fa17 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:33.982803 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658dd1e7-f482-4a5e-82fc-84a257b3fa17 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:33.983634 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658dd1e7-f482-4a5e-82fc-84a257b3fa17 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:33.984647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658dd1e7-f482-4a5e-82fc-84a257b3fa17 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:33.986894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658dd1e7-f482-4a5e-82fc-84a257b3fa17 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:33.990618 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658dd1e7-f482-4a5e-82fc-84a257b3fa17 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:34.018764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:34.037150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 378/753] 10.222.0.47 () {56 vars in 1133 bytes} [Sun Aug 21 01:05:33 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:34.037150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658dd1e7-f482-4a5e-82fc-84a257b3fa17 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:34.114203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfc395e-1bc6-43d5-ab5f-badfea50bb55 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:34.212499 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658dd1e7-f482-4a5e-82fc-84a257b3fa17 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:34.217003 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce4cca9-4001-49b4-955e-8b2eb817cef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:34.217567 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce4cca9-4001-49b4-955e-8b2eb817cef9 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:34.217846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce4cca9-4001-49b4-955e-8b2eb817cef9 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:34.218120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce4cca9-4001-49b4-955e-8b2eb817cef9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:34.224337 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:34.234650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce4cca9-4001-49b4-955e-8b2eb817cef9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:34.237248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce4cca9-4001-49b4-955e-8b2eb817cef9 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:34.257228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-0ce4cca9-4001-49b4-955e-8b2eb817cef9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:34.257228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:34.257228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:34.257228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:34.257228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:34.257228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:34.257228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:34.257228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:34.257228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:34.257228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:34.257228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:34.257228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:34.257228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:34.257228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:34.257228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:34.257228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:34.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:34.259584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:34.259584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:34.259584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:34.259808 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cd311a-0e7e-431c-ac8f-592b920f44ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:34.260275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cd311a-0e7e-431c-ac8f-592b920f44ff None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:34.260640 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cd311a-0e7e-431c-ac8f-592b920f44ff None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:34.260994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cd311a-0e7e-431c-ac8f-592b920f44ff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:34.267061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 376/754] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 354 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:34.267513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce4cca9-4001-49b4-955e-8b2eb817cef9 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:34.278321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cd311a-0e7e-431c-ac8f-592b920f44ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:34.279818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cd311a-0e7e-431c-ac8f-592b920f44ff None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:34.300919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-56cd311a-0e7e-431c-ac8f-592b920f44ff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:34.300919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:34.300919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:34.300919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:34.300919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:34.300919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:34.300919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:34.300919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:34.300919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:34.300919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:34.300919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:34.300919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:34.300919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:34.300919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:34.300919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:34.300919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:34.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:34.303380 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:34.303380 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:34.303380 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:34.303380 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 379/755] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:34.303380 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cd311a-0e7e-431c-ac8f-592b920f44ff None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:34.401222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:34.408945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce4cca9-4001-49b4-955e-8b2eb817cef9 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:34.439214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cd311a-0e7e-431c-ac8f-592b920f44ff None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:34.449808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fcc1ec-c811-4cfe-8281-3702d8ff1854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:34.450448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fcc1ec-c811-4cfe-8281-3702d8ff1854 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:34.450722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fcc1ec-c811-4cfe-8281-3702d8ff1854 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:34.450994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fcc1ec-c811-4cfe-8281-3702d8ff1854 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:34.451576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fcc1ec-c811-4cfe-8281-3702d8ff1854 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:34.453116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fcc1ec-c811-4cfe-8281-3702d8ff1854 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-72fcc1ec-c811-4cfe-8281-3702d8ff1854 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:34.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:34.466961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:34.466961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:34.466961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:34.466961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:34.466961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:34.466961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:34.466961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:05:34.466961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:05:34.466961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:05:34.466961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:05:34.466961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:34.466961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:34.466961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:34.466961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:34.466961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:34.466961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:34.467801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:34.467801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:34.467801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:34.467801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:34.467801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:34.467801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:34.467801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:34.467801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:34.467801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:34.467801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:34.467801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:05:34.467801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:34.467801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 21 01:05:34.467801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:34.467801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 377/756] 10.222.0.47 () {56 vars in 1129 bytes} [Sun Aug 21 01:05:34 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:34.468644 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fcc1ec-c811-4cfe-8281-3702d8ff1854 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:34.483405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7514a61-fc88-42f4-9799-74c224df0e4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:34.484154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7514a61-fc88-42f4-9799-74c224df0e4a None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:34.484658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7514a61-fc88-42f4-9799-74c224df0e4a None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:34.485026 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7514a61-fc88-42f4-9799-74c224df0e4a None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:34.486139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7514a61-fc88-42f4-9799-74c224df0e4a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:34.486511 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7514a61-fc88-42f4-9799-74c224df0e4a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:34.488307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7514a61-fc88-42f4-9799-74c224df0e4a None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:34.505199 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 380/757] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:34.506131 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7514a61-fc88-42f4-9799-74c224df0e4a None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:34.529282 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:34.595948 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fcc1ec-c811-4cfe-8281-3702d8ff1854 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:34.635490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7514a61-fc88-42f4-9799-74c224df0e4a None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:34.649793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acfa2fd-1ac8-4439-9862-08de3f1b44cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:34.650666 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acfa2fd-1ac8-4439-9862-08de3f1b44cd None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:34.651406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acfa2fd-1ac8-4439-9862-08de3f1b44cd None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:34.651697 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acfa2fd-1ac8-4439-9862-08de3f1b44cd None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:34.652781 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acfa2fd-1ac8-4439-9862-08de3f1b44cd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:34.653088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acfa2fd-1ac8-4439-9862-08de3f1b44cd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:34.655127 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acfa2fd-1ac8-4439-9862-08de3f1b44cd None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:34.671597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 378/758] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:34.672435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecb3d13-adbe-4995-9e7e-d259afb33c7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:34.673101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecb3d13-adbe-4995-9e7e-d259afb33c7b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:34.673380 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecb3d13-adbe-4995-9e7e-d259afb33c7b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:34.673647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecb3d13-adbe-4995-9e7e-d259afb33c7b None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:34.674265 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecb3d13-adbe-4995-9e7e-d259afb33c7b None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:34.674588 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecb3d13-adbe-4995-9e7e-d259afb33c7b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:34.675934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecb3d13-adbe-4995-9e7e-d259afb33c7b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:34.676895 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acfa2fd-1ac8-4439-9862-08de3f1b44cd None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:34.681751 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 381/759] 10.222.0.47 () {56 vars in 1124 bytes} [Sun Aug 21 01:05:34 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:34.682613 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecb3d13-adbe-4995-9e7e-d259afb33c7b None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:34.709291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:34.778742 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:34.794992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acfa2fd-1ac8-4439-9862-08de3f1b44cd None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:34.797794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecb3d13-adbe-4995-9e7e-d259afb33c7b None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:34.834072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f08e6dc-d26c-4457-aa92-05bb37675ed8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:34.834702 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f08e6dc-d26c-4457-aa92-05bb37675ed8 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:34.834976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f08e6dc-d26c-4457-aa92-05bb37675ed8 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:34.835240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f08e6dc-d26c-4457-aa92-05bb37675ed8 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:34.836227 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c833f68c-6d39-4a02-a6cd-cfb9f472d8c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:34.836863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c833f68c-6d39-4a02-a6cd-cfb9f472d8c7 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:34.837137 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c833f68c-6d39-4a02-a6cd-cfb9f472d8c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:34.837408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c833f68c-6d39-4a02-a6cd-cfb9f472d8c7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:34.843093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f08e6dc-d26c-4457-aa92-05bb37675ed8 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:34.844415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c833f68c-6d39-4a02-a6cd-cfb9f472d8c7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:34.844532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f08e6dc-d26c-4457-aa92-05bb37675ed8 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:34.845833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c833f68c-6d39-4a02-a6cd-cfb9f472d8c7 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:34.856013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-8f08e6dc-d26c-4457-aa92-05bb37675ed8 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 21 01:05:34.856013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:34.856013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:34.856013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:34.856013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:34.856013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:34.856013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:34.856013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:34.856013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:34.856013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:34.856013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:34.856013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:34.856013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:34.856013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:34.856013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:34.856013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:34.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:34.857972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:34.857972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:34.857972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:34.857972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:34.857972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:34.857972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:34.857972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:34.857972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:34.857972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 21 01:05:34.857972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:34.858514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-c833f68c-6d39-4a02-a6cd-cfb9f472d8c7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:34.858514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:34.858514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:34.858514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:34.858514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:34.858514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:34.858514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:34.858514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:34.858514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:34.858514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:34.858514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:34.858514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:34.858514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:34.858514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:34.858514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:34.859295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:34.861231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:34.861231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:34.861231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:34.861231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:34.861231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:34.861231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:34.861231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:34.861231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:34.863114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 379/760] 10.222.0.47 () {56 vars in 1117 bytes} [Sun Aug 21 01:05:34 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:34.863114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f08e6dc-d26c-4457-aa92-05bb37675ed8 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:34.863401 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 382/761] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:34.864570 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d436856-a333-4855-9496-98f01514614d None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:34.864854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d436856-a333-4855-9496-98f01514614d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:34.865126 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d436856-a333-4855-9496-98f01514614d None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:34.870746 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 383/762] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:34.870746 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d436856-a333-4855-9496-98f01514614d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:34.960403 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:34.980780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f08e6dc-d26c-4457-aa92-05bb37675ed8 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:34.992774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d436856-a333-4855-9496-98f01514614d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:35.029632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67c4f02-20cb-4154-bb7d-15be616b40ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:35.030351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67c4f02-20cb-4154-bb7d-15be616b40ff None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:35.030722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67c4f02-20cb-4154-bb7d-15be616b40ff None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:35.030997 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67c4f02-20cb-4154-bb7d-15be616b40ff None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:35.031624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67c4f02-20cb-4154-bb7d-15be616b40ff None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:35.031923 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67c4f02-20cb-4154-bb7d-15be616b40ff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:35.033332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67c4f02-20cb-4154-bb7d-15be616b40ff None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:35.044350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 380/763] 10.222.0.47 () {56 vars in 1124 bytes} [Sun Aug 21 01:05:34 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:35.044350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67c4f02-20cb-4154-bb7d-15be616b40ff None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:35.047650 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e52ffb5-4c9b-4753-8cd6-6cd91fda0f85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:35.048863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e52ffb5-4c9b-4753-8cd6-6cd91fda0f85 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:35.048863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e52ffb5-4c9b-4753-8cd6-6cd91fda0f85 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:35.050129 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e52ffb5-4c9b-4753-8cd6-6cd91fda0f85 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:35.050129 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e52ffb5-4c9b-4753-8cd6-6cd91fda0f85 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:35.050402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e52ffb5-4c9b-4753-8cd6-6cd91fda0f85 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:35.051827 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e52ffb5-4c9b-4753-8cd6-6cd91fda0f85 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:35.067377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:35.071440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 384/764] 10.222.0.47 () {56 vars in 1149 bytes} [Sun Aug 21 01:05:34 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:35.072957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e444d303-b321-4925-89f2-4e33a774f02a None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:35.077197 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e444d303-b321-4925-89f2-4e33a774f02a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:35.077197 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e444d303-b321-4925-89f2-4e33a774f02a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:35.098114 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-e444d303-b321-4925-89f2-4e33a774f02a None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:35.099371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e444d303-b321-4925-89f2-4e33a774f02a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:35.180587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67c4f02-20cb-4154-bb7d-15be616b40ff None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:35.194098 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 385/765] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:35 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:35.205086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e444d303-b321-4925-89f2-4e33a774f02a None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:35.230742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31a7a67-9def-4518-aff9-f139b36deb2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:35.231817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31a7a67-9def-4518-aff9-f139b36deb2f None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:35.232322 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31a7a67-9def-4518-aff9-f139b36deb2f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:35.232801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31a7a67-9def-4518-aff9-f139b36deb2f None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:35.233758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31a7a67-9def-4518-aff9-f139b36deb2f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:35.234249 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31a7a67-9def-4518-aff9-f139b36deb2f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:35.235998 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31a7a67-9def-4518-aff9-f139b36deb2f None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:35.264084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 381/766] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:35.264084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31a7a67-9def-4518-aff9-f139b36deb2f None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:35.268033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:35.336855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e444d303-b321-4925-89f2-4e33a774f02a None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:35.381831 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:35.389626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31a7a67-9def-4518-aff9-f139b36deb2f None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:35.391325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e857f9-7b28-4d9a-8f06-496e6a892537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:35.392503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e857f9-7b28-4d9a-8f06-496e6a892537 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:35.394522 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e857f9-7b28-4d9a-8f06-496e6a892537 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:35.394522 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e857f9-7b28-4d9a-8f06-496e6a892537 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:35.395461 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e857f9-7b28-4d9a-8f06-496e6a892537 None admin] RBAC: Adding query filter params (interface=public, service_id=a31891285bd14b919e4b12e9297546a0, region_id=RegionOne) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:35.395990 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e857f9-7b28-4d9a-8f06-496e6a892537 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:35.400345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e857f9-7b28-4d9a-8f06-496e6a892537 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:35.418754 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 386/767] 10.222.0.47 () {56 vars in 1251 bytes} [Sun Aug 21 01:05:35 2022] GET /identity/v3/endpoints?service_id=a31891285bd14b919e4b12e9297546a0&interface=public®ion_id=RegionOne => generated 194 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:35.420335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e857f9-7b28-4d9a-8f06-496e6a892537 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:35.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca3387b-45c8-4523-ba8e-c15990dddb9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:35.431787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca3387b-45c8-4523-ba8e-c15990dddb9c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:35.432278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca3387b-45c8-4523-ba8e-c15990dddb9c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:35.432753 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca3387b-45c8-4523-ba8e-c15990dddb9c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:35.507246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca3387b-45c8-4523-ba8e-c15990dddb9c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:35.508736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca3387b-45c8-4523-ba8e-c15990dddb9c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:35.511505 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:35.586619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e857f9-7b28-4d9a-8f06-496e6a892537 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:35.624923 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 382/768] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:35 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:05:35.625779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca3387b-45c8-4523-ba8e-c15990dddb9c None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:35.632802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9563f53-43f3-4f2b-b66d-fec97a793c9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:35.633481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9563f53-43f3-4f2b-b66d-fec97a793c9a None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:35.633761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9563f53-43f3-4f2b-b66d-fec97a793c9a None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:35.634034 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9563f53-43f3-4f2b-b66d-fec97a793c9a None admin] PATH_INFO: `/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/fabdcdc25d1d4ef6bc66bd6cadecaaf8/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:35.635981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9563f53-43f3-4f2b-b66d-fec97a793c9a None admin] RBAC: Authorizing `identity:create_grant(project_id=75a3bf3f3a4244d18fde47d492657db4, user_id=fabdcdc25d1d4ef6bc66bd6cadecaaf8, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:35.637482 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9563f53-43f3-4f2b-b66d-fec97a793c9a None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:35.712938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 387/769] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:05:35 2022] PUT /identity/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/fabdcdc25d1d4ef6bc66bd6cadecaaf8/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:05:35.715111 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9563f53-43f3-4f2b-b66d-fec97a793c9a None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:35.749598 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca3387b-45c8-4523-ba8e-c15990dddb9c None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:35.753299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:35.792513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:35.797646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a3e5c1-139e-40e6-840d-3c8556329e0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:35.798279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a3e5c1-139e-40e6-840d-3c8556329e0a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:35.798553 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a3e5c1-139e-40e6-840d-3c8556329e0a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:35.798849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a3e5c1-139e-40e6-840d-3c8556329e0a None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:35.799505 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a3e5c1-139e-40e6-840d-3c8556329e0a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:35.800951 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a3e5c1-139e-40e6-840d-3c8556329e0a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:35.807278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 383/770] 10.222.0.47 () {56 vars in 1087 bytes} [Sun Aug 21 01:05:35 2022] GET /identity/v3/services => generated 772 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:35.808263 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a3e5c1-139e-40e6-840d-3c8556329e0a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:35.867599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9563f53-43f3-4f2b-b66d-fec97a793c9a None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:35.930900 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559c4b9a-2eb6-4dc6-be6a-fa02847ef398 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:35.931969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559c4b9a-2eb6-4dc6-be6a-fa02847ef398 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:35.932266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559c4b9a-2eb6-4dc6-be6a-fa02847ef398 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:35.932537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559c4b9a-2eb6-4dc6-be6a-fa02847ef398 None admin] PATH_INFO: `/v3/services/network` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:35.933132 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559c4b9a-2eb6-4dc6-be6a-fa02847ef398 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:35.935087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559c4b9a-2eb6-4dc6-be6a-fa02847ef398 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-559c4b9a-2eb6-4dc6-be6a-fa02847ef398 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:35.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:35.946458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:35.946458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:35.946458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:35.946458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:35.946458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:35.946458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:35.946458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:05:35.946458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:05:35.946458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:05:35.946458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:05:35.946458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:35.946458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:35.946458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:35.946458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:35.946458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:35.946458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:35.947314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:35.947314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:35.947314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:35.947314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:35.947314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:35.947314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:35.947314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:35.947314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:35.947314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:35.947314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:35.947314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:05:35.947314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:35.947314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 21 01:05:35.947314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:35.947314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 388/771] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:35 2022] GET /identity/v3/services/network => generated 88 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:35.948686 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d6218a-4862-490b-9a86-3acf41626d66 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:35.948686 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d6218a-4862-490b-9a86-3acf41626d66 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:35.948857 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d6218a-4862-490b-9a86-3acf41626d66 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:35.950183 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 389/772] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:35.954942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d6218a-4862-490b-9a86-3acf41626d66 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:35.959763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a3e5c1-139e-40e6-840d-3c8556329e0a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:36.001262 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90ecb74-e333-4c81-97c2-c84d0f585fb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:36.002260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90ecb74-e333-4c81-97c2-c84d0f585fb3 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:36.003847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90ecb74-e333-4c81-97c2-c84d0f585fb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:36.004391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90ecb74-e333-4c81-97c2-c84d0f585fb3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:36.016239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90ecb74-e333-4c81-97c2-c84d0f585fb3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:36.019404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90ecb74-e333-4c81-97c2-c84d0f585fb3 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:36.043626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-a90ecb74-e333-4c81-97c2-c84d0f585fb3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:36.043626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:36.043626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:36.043626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:36.043626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:36.043626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:36.043626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:36.043626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:36.043626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:36.043626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:36.043626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:36.043626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:36.043626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:36.043626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:36.043626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:36.043626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:36.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:36.045658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:36.045658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:36.045658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:36.047371 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 384/773] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:36.053050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbaab63-dacb-4335-8acd-2ae345751e15 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:36.053581 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbaab63-dacb-4335-8acd-2ae345751e15 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:36.054010 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbaab63-dacb-4335-8acd-2ae345751e15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:36.078840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-4dbaab63-dacb-4335-8acd-2ae345751e15 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:36.080365 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-4dbaab63-dacb-4335-8acd-2ae345751e15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:36.088676 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d6218a-4862-490b-9a86-3acf41626d66 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:36.101385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:36.110597 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:36.130847 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c99168d-af23-41f7-96aa-06999efd66a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:36.131711 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c99168d-af23-41f7-96aa-06999efd66a9 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:36.132205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c99168d-af23-41f7-96aa-06999efd66a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:36.132646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c99168d-af23-41f7-96aa-06999efd66a9 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:36.133493 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c99168d-af23-41f7-96aa-06999efd66a9 None admin] RBAC: Adding query filter params (name=network) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:36.133942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c99168d-af23-41f7-96aa-06999efd66a9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:36.135521 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c99168d-af23-41f7-96aa-06999efd66a9 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:36.141231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 390/774] 10.222.0.47 () {56 vars in 1112 bytes} [Sun Aug 21 01:05:35 2022] GET /identity/v3/services?name=network => generated 124 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:36.142351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c99168d-af23-41f7-96aa-06999efd66a9 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:36.175634 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 385/775] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:36 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:36.177318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbaab63-dacb-4335-8acd-2ae345751e15 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:36.323313 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c99168d-af23-41f7-96aa-06999efd66a9 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:36.386201 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbaab63-dacb-4335-8acd-2ae345751e15 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:36.396368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993df28b-3144-4a5f-9052-a0a9a712f3d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:36.397043 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993df28b-3144-4a5f-9052-a0a9a712f3d3 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:36.397326 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993df28b-3144-4a5f-9052-a0a9a712f3d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:36.397597 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993df28b-3144-4a5f-9052-a0a9a712f3d3 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:36.398243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993df28b-3144-4a5f-9052-a0a9a712f3d3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:36.398554 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993df28b-3144-4a5f-9052-a0a9a712f3d3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:36.399915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993df28b-3144-4a5f-9052-a0a9a712f3d3 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:36.429298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 391/776] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:36.430262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993df28b-3144-4a5f-9052-a0a9a712f3d3 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:36.465138 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e43ae3e-b00d-45a0-a52f-397bdc2842a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:36.466241 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e43ae3e-b00d-45a0-a52f-397bdc2842a9 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:36.466798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e43ae3e-b00d-45a0-a52f-397bdc2842a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:36.467077 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e43ae3e-b00d-45a0-a52f-397bdc2842a9 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:36.468090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e43ae3e-b00d-45a0-a52f-397bdc2842a9 None admin] RBAC: Adding query filter params (type=network) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:36.468423 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e43ae3e-b00d-45a0-a52f-397bdc2842a9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:36.470529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e43ae3e-b00d-45a0-a52f-397bdc2842a9 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:36.479029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 386/777] 10.222.0.47 () {56 vars in 1112 bytes} [Sun Aug 21 01:05:36 2022] GET /identity/v3/services?type=network => generated 350 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:36.479886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e43ae3e-b00d-45a0-a52f-397bdc2842a9 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:36.490350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:36.504398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:36.578817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993df28b-3144-4a5f-9052-a0a9a712f3d3 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:36.601240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e43ae3e-b00d-45a0-a52f-397bdc2842a9 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:36.633759 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bda3d27-e577-49d2-8008-bb522ac6d880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:36.634670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bda3d27-e577-49d2-8008-bb522ac6d880 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:36.635062 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bda3d27-e577-49d2-8008-bb522ac6d880 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:36.635393 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bda3d27-e577-49d2-8008-bb522ac6d880 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:36.638407 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ea1848-ace3-4329-a696-30948f867d2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:36.639036 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ea1848-ace3-4329-a696-30948f867d2f None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:36.639310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ea1848-ace3-4329-a696-30948f867d2f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:36.639577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ea1848-ace3-4329-a696-30948f867d2f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:36.644831 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bda3d27-e577-49d2-8008-bb522ac6d880 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:36.646809 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bda3d27-e577-49d2-8008-bb522ac6d880 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:36.651512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ea1848-ace3-4329-a696-30948f867d2f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:36.656482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-e3ea1848-ace3-4329-a696-30948f867d2f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:05:36.656482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:36.656482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:36.656482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:36.656482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:36.656482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:36.656482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:36.656482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:36.656482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:36.656482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:36.656482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:36.656482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:36.656482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:36.656482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:36.656482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:36.656482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:36.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:36.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:36.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:36.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:36.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:36.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:36.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:36.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:36.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:36.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:36.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:36.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:36.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:36.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:36.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:36.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:36.660693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:36.660693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:36.660693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:36.660693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:36.660693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:36.660693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:36.660693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:36.660693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:36.660693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:36.660693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:36.660693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:05:36.660693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:36.662200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-9bda3d27-e577-49d2-8008-bb522ac6d880 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:36.662200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:36.662200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:36.662200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:36.662200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:36.662200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:36.662200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:36.662200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:36.662200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:36.662200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:36.662200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:36.662200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:36.662200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:36.662200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:36.662200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:36.664120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:36.674683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:36.674683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:36.674683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:36.674683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:36.674683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:36.674683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:36.674683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:36.674683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:36.676049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 387/779] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:05:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:36.676049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ea1848-ace3-4329-a696-30948f867d2f None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:36.676425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 392/779] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:36.676425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bda3d27-e577-49d2-8008-bb522ac6d880 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:36.780871 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:36.810431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ea1848-ace3-4329-a696-30948f867d2f None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:36.810656 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bda3d27-e577-49d2-8008-bb522ac6d880 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:36.812357 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:36.853958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb9876f-d5ac-4970-8645-a9c268a628e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:36.854940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb9876f-d5ac-4970-8645-a9c268a628e9 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:36.855481 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb9876f-d5ac-4970-8645-a9c268a628e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:36.855815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb9876f-d5ac-4970-8645-a9c268a628e9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:36.856685 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb9876f-d5ac-4970-8645-a9c268a628e9 None admin] RBAC: Adding query filter params (interface=public, service_id=b548786a151641b584f8fcbe23e5c0fa, region_id=RegionOne) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:36.857055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb9876f-d5ac-4970-8645-a9c268a628e9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:36.858559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb9876f-d5ac-4970-8645-a9c268a628e9 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:36.861487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad7a054-309e-499f-b5df-2032d847504c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:36.865414 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad7a054-309e-499f-b5df-2032d847504c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:36.866222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad7a054-309e-499f-b5df-2032d847504c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:36.866677 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad7a054-309e-499f-b5df-2032d847504c None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:36.867524 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad7a054-309e-499f-b5df-2032d847504c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:36.868002 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad7a054-309e-499f-b5df-2032d847504c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:36.869648 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad7a054-309e-499f-b5df-2032d847504c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:36.872656 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 388/780] 10.222.0.47 () {56 vars in 1251 bytes} [Sun Aug 21 01:05:36 2022] GET /identity/v3/endpoints?service_id=b548786a151641b584f8fcbe23e5c0fa&interface=public®ion_id=RegionOne => generated 194 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:36.873718 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb9876f-d5ac-4970-8645-a9c268a628e9 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:36.888744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 393/781] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:05:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:36.889461 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad7a054-309e-499f-b5df-2032d847504c None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:36.996911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb9876f-d5ac-4970-8645-a9c268a628e9 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:37.046166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8641ba-c8d8-4a7f-ad83-32a3d6d1cc42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:37.047397 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8641ba-c8d8-4a7f-ad83-32a3d6d1cc42 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:37.048296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8641ba-c8d8-4a7f-ad83-32a3d6d1cc42 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:37.049046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8641ba-c8d8-4a7f-ad83-32a3d6d1cc42 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:37.050684 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8641ba-c8d8-4a7f-ad83-32a3d6d1cc42 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:37.051667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8641ba-c8d8-4a7f-ad83-32a3d6d1cc42 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:37.053704 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8641ba-c8d8-4a7f-ad83-32a3d6d1cc42 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:37.054336 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad7a054-309e-499f-b5df-2032d847504c None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:37.064714 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:37.078954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 389/782] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:37.078954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8641ba-c8d8-4a7f-ad83-32a3d6d1cc42 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:37.103893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2452b57f-6fb5-4666-a5fa-8379784302f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:37.105917 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2452b57f-6fb5-4666-a5fa-8379784302f9 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:37.106696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2452b57f-6fb5-4666-a5fa-8379784302f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:37.107486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2452b57f-6fb5-4666-a5fa-8379784302f9 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:37.108898 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2452b57f-6fb5-4666-a5fa-8379784302f9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:37.112183 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2452b57f-6fb5-4666-a5fa-8379784302f9 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:37.124938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 394/783] 10.222.0.47 () {56 vars in 1087 bytes} [Sun Aug 21 01:05:36 2022] GET /identity/v3/services => generated 772 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:37.126713 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2452b57f-6fb5-4666-a5fa-8379784302f9 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:37.203582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8641ba-c8d8-4a7f-ad83-32a3d6d1cc42 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:37.213484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:37.287692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f04ae64-11d5-4aa9-874d-55b399a86350 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:37.288676 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f04ae64-11d5-4aa9-874d-55b399a86350 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:37.288956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f04ae64-11d5-4aa9-874d-55b399a86350 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:37.289233 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f04ae64-11d5-4aa9-874d-55b399a86350 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:37.321909 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f04ae64-11d5-4aa9-874d-55b399a86350 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:37.323395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f04ae64-11d5-4aa9-874d-55b399a86350 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:37.355458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-9f04ae64-11d5-4aa9-874d-55b399a86350 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:37.355458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:37.355458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:37.355458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:37.355458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:37.355458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:37.355458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:37.355458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:37.355458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:37.355458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:37.355458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:37.355458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:37.355458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:37.355458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:37.355458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:37.355458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:37.357459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:37.358385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:37.358385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:37.358385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:37.358575 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2452b57f-6fb5-4666-a5fa-8379784302f9 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:37.360027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 390/784] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:37.368057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f04ae64-11d5-4aa9-874d-55b399a86350 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:37.413840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a20b2d-14cc-456b-a454-42695e32f4f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:37.414454 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a20b2d-14cc-456b-a454-42695e32f4f7 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:37.414725 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a20b2d-14cc-456b-a454-42695e32f4f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:37.414988 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a20b2d-14cc-456b-a454-42695e32f4f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:37.415702 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a20b2d-14cc-456b-a454-42695e32f4f7 None admin] RBAC: Adding query filter params (role.id=c2d8bf7283e14857879d03d1f39c0165, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=d7c65195ff8d42958fa54b5f083bf8da) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:37.416683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a20b2d-14cc-456b-a454-42695e32f4f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:37.418568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a20b2d-14cc-456b-a454-42695e32f4f7 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:37.424950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 395/785] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:37 2022] GET /identity/v3/role_assignments?user.id=d7c65195ff8d42958fa54b5f083bf8da&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=c2d8bf7283e14857879d03d1f39c0165 => generated 259 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:37.461791 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:37.547332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f04ae64-11d5-4aa9-874d-55b399a86350 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:37.600865 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31f440f-9c9a-4732-b4cc-d0cad449a934 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:37.601496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31f440f-9c9a-4732-b4cc-d0cad449a934 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:37.601771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31f440f-9c9a-4732-b4cc-d0cad449a934 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:37.602044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31f440f-9c9a-4732-b4cc-d0cad449a934 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:37.602700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31f440f-9c9a-4732-b4cc-d0cad449a934 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:37.602996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31f440f-9c9a-4732-b4cc-d0cad449a934 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:37.604465 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31f440f-9c9a-4732-b4cc-d0cad449a934 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:37.618274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 391/786] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:37.623981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a20b2d-14cc-456b-a454-42695e32f4f7 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:37.731957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:37.744599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a20b2d-14cc-456b-a454-42695e32f4f7 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:37.787518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26aba868-ea52-47f7-802a-483b0dbb1b12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:37.788185 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26aba868-ea52-47f7-802a-483b0dbb1b12 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:37.788461 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26aba868-ea52-47f7-802a-483b0dbb1b12 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:37.788729 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26aba868-ea52-47f7-802a-483b0dbb1b12 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:37.796391 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26aba868-ea52-47f7-802a-483b0dbb1b12 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:37.797880 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26aba868-ea52-47f7-802a-483b0dbb1b12 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:37.812710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-26aba868-ea52-47f7-802a-483b0dbb1b12 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 21 01:05:37.812710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:37.812710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:37.812710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:37.812710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:37.812710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:37.812710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:37.812710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:37.812710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:37.812710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:37.812710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:37.812710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:37.812710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:37.812710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:37.812710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:37.812710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:37.813738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:37.814837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:37.814837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:37.814837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:37.814837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:37.814837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:37.814837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:37.814837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:37.814837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:37.814837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 21 01:05:37.814837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:37.819297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 396/787] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:05:37 2022] GET /identity/v3/users/swift => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:37.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31f440f-9c9a-4732-b4cc-d0cad449a934 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:37.936763 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aadbdbc-0033-49e6-bf06-b2756517ac06 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:37.936960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aadbdbc-0033-49e6-bf06-b2756517ac06 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:37.937224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aadbdbc-0033-49e6-bf06-b2756517ac06 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:37.937604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31f440f-9c9a-4732-b4cc-d0cad449a934 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:37.939554 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 397/788] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:37.944584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea49c63c-ecbb-4ab9-8050-0a743dfa70e6 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:37.944869 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea49c63c-ecbb-4ab9-8050-0a743dfa70e6 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:37.945144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea49c63c-ecbb-4ab9-8050-0a743dfa70e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:37.952439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:37.979181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb48a66c-8b3a-419d-8c40-53796dd65da5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:37.980284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb48a66c-8b3a-419d-8c40-53796dd65da5 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:37.980838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb48a66c-8b3a-419d-8c40-53796dd65da5 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:37.981110 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-ea49c63c-ecbb-4ab9-8050-0a743dfa70e6 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:37.981534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb48a66c-8b3a-419d-8c40-53796dd65da5 None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:37.982447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ea49c63c-ecbb-4ab9-8050-0a743dfa70e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:37.982564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb48a66c-8b3a-419d-8c40-53796dd65da5 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:37.983056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb48a66c-8b3a-419d-8c40-53796dd65da5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:37.984891 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb48a66c-8b3a-419d-8c40-53796dd65da5 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:38.010289 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 392/789] 10.222.0.47 () {56 vars in 1135 bytes} [Sun Aug 21 01:05:37 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:38.012028 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff35aeff-8e13-437a-95d0-bd08832463da None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:38.012488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff35aeff-8e13-437a-95d0-bd08832463da None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:38.012915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff35aeff-8e13-437a-95d0-bd08832463da None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:38.014522 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 393/790] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:38.015659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff35aeff-8e13-437a-95d0-bd08832463da None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:38.062558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 398/791] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:37 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:38.063928 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c594d749-8b81-402d-b133-d83062dcccde None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:38.064242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c594d749-8b81-402d-b133-d83062dcccde None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:38.064517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c594d749-8b81-402d-b133-d83062dcccde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:38.097724 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-c594d749-8b81-402d-b133-d83062dcccde None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:38.099085 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-c594d749-8b81-402d-b133-d83062dcccde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:38.137461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff35aeff-8e13-437a-95d0-bd08832463da None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:38.175335 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e30898c-8250-4fc4-9c03-5e394710df9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:38.176322 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e30898c-8250-4fc4-9c03-5e394710df9c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:38.176759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e30898c-8250-4fc4-9c03-5e394710df9c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:38.177188 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e30898c-8250-4fc4-9c03-5e394710df9c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:38.188122 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e30898c-8250-4fc4-9c03-5e394710df9c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:38.190028 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e30898c-8250-4fc4-9c03-5e394710df9c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:38.206144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-2e30898c-8250-4fc4-9c03-5e394710df9c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:38.206144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:38.206144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:38.206144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:38.206144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:38.206144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:38.206144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:38.206144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:38.206144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:38.206144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:38.206144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:38.206144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:38.206144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:38.206144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:38.206144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:38.206144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:38.207120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:38.208020 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:38.208020 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:38.208020 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:38.210439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 394/792] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:38.216312 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e30898c-8250-4fc4-9c03-5e394710df9c None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:38.256027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:38.303631 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 399/793] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:38 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:38.370193 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e30898c-8250-4fc4-9c03-5e394710df9c None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:38.371002 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c594d749-8b81-402d-b133-d83062dcccde None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:38.404454 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:38.411342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2088b6-654e-4a48-ad83-87bc6977077f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:38.412246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2088b6-654e-4a48-ad83-87bc6977077f None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:38.412527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2088b6-654e-4a48-ad83-87bc6977077f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:38.412797 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2088b6-654e-4a48-ad83-87bc6977077f None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:38.413441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2088b6-654e-4a48-ad83-87bc6977077f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:38.413739 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2088b6-654e-4a48-ad83-87bc6977077f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:38.415109 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2088b6-654e-4a48-ad83-87bc6977077f None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:38.429668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 395/794] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:38.435299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2088b6-654e-4a48-ad83-87bc6977077f None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:38.533013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c594d749-8b81-402d-b133-d83062dcccde None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:38.591791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d21482c-cc5b-4552-99f3-bcdf6c44e5ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:38.592672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d21482c-cc5b-4552-99f3-bcdf6c44e5ae None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:38.593159 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d21482c-cc5b-4552-99f3-bcdf6c44e5ae None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:38.593645 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d21482c-cc5b-4552-99f3-bcdf6c44e5ae None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:38.613443 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d21482c-cc5b-4552-99f3-bcdf6c44e5ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:38.619297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-3d21482c-cc5b-4552-99f3-bcdf6c44e5ae None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:05:38.619297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:38.619297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:38.619297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:38.619297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:38.619297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:38.619297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:38.619297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:38.619297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:38.619297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:38.619297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:38.619297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:38.619297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:38.619297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:38.619297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:38.619297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:38.620279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:38.620279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:38.620279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:38.620279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:38.620279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:38.620279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:38.620279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:38.620279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:38.620279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:38.620279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:38.620279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:38.620279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:38.620279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:38.620279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:38.620279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:38.620279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:38.621360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:38.621360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:38.621360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:38.621360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:38.621360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:38.621360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:38.621360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:38.621360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:38.621360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:38.621360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:38.621360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:05:38.621360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:38.621360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 400/795] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:05:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:38.631524 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d21482c-cc5b-4552-99f3-bcdf6c44e5ae None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:38.666336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:38.701983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2088b6-654e-4a48-ad83-87bc6977077f None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:38.744394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:38.752256 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52affc3-80f7-4fe9-98f5-7d218df3d622 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:38.753642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52affc3-80f7-4fe9-98f5-7d218df3d622 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:38.754133 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52affc3-80f7-4fe9-98f5-7d218df3d622 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:38.754924 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52affc3-80f7-4fe9-98f5-7d218df3d622 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:38.764572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52affc3-80f7-4fe9-98f5-7d218df3d622 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:38.767099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52affc3-80f7-4fe9-98f5-7d218df3d622 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:38.780121 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d21482c-cc5b-4552-99f3-bcdf6c44e5ae None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:38.785643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-a52affc3-80f7-4fe9-98f5-7d218df3d622 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:38.785643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:38.785643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:38.785643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:38.785643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:38.785643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:38.785643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:38.785643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:38.785643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:38.785643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:38.785643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:38.785643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:38.785643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:38.785643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:38.785643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:38.786570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:38.787457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:38.787457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:38.787457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:38.787457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:38.787457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:38.787457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:38.787457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:38.787457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:38.795853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 396/796] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 357 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:38.797227 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52affc3-80f7-4fe9-98f5-7d218df3d622 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:38.834215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ce471e-3142-4941-bf02-ca0eb5e91ecc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:38.834871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ce471e-3142-4941-bf02-ca0eb5e91ecc None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:38.835146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ce471e-3142-4941-bf02-ca0eb5e91ecc None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:38.835463 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ce471e-3142-4941-bf02-ca0eb5e91ecc None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:38.836415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ce471e-3142-4941-bf02-ca0eb5e91ecc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:38.836713 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ce471e-3142-4941-bf02-ca0eb5e91ecc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:38.838782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ce471e-3142-4941-bf02-ca0eb5e91ecc None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:38.850868 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 401/797] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:05:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:38.851963 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ce471e-3142-4941-bf02-ca0eb5e91ecc None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:38.937493 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52affc3-80f7-4fe9-98f5-7d218df3d622 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:38.990177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ce471e-3142-4941-bf02-ca0eb5e91ecc None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:38.999380 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de26b7e-180e-4659-ba32-79b0e859934a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:39.000913 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de26b7e-180e-4659-ba32-79b0e859934a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:39.001662 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de26b7e-180e-4659-ba32-79b0e859934a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:39.002919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de26b7e-180e-4659-ba32-79b0e859934a None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:39.004317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de26b7e-180e-4659-ba32-79b0e859934a None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:39.007191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de26b7e-180e-4659-ba32-79b0e859934a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-6de26b7e-180e-4659-ba32-79b0e859934a None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:39.025511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:39.027309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:39.027309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:39.027309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:39.027309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:39.027309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:39.027309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:39.027309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:05:39.027309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:05:39.027309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:05:39.027309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:05:39.027309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:39.027309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:39.027309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:39.027309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:39.027309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:39.027309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:39.028956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:39.028956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:39.028956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:39.028956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:39.028956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:39.028956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:39.028956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:39.028956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:39.028956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:39.028956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:39.028956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:05:39.028956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:39.028956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 21 01:05:39.028956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:39.028956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 397/798] 10.222.0.47 () {56 vars in 1129 bytes} [Sun Aug 21 01:05:38 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:39.038982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de26b7e-180e-4659-ba32-79b0e859934a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:39.040800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daed5809-237d-48a3-aa7c-fffccd553c80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:39.042183 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daed5809-237d-48a3-aa7c-fffccd553c80 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:39.042943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daed5809-237d-48a3-aa7c-fffccd553c80 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:39.043741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daed5809-237d-48a3-aa7c-fffccd553c80 None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:39.045375 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daed5809-237d-48a3-aa7c-fffccd553c80 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:39.046180 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daed5809-237d-48a3-aa7c-fffccd553c80 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:39.049154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daed5809-237d-48a3-aa7c-fffccd553c80 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:39.052828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:39.073431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 402/799] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:39.075192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daed5809-237d-48a3-aa7c-fffccd553c80 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:39.181206 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:39.203143 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daed5809-237d-48a3-aa7c-fffccd553c80 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:39.249627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8443468-a026-4333-b112-87ba65b465ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:39.250232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8443468-a026-4333-b112-87ba65b465ff None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:39.250500 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8443468-a026-4333-b112-87ba65b465ff None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:39.250770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8443468-a026-4333-b112-87ba65b465ff None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:39.251397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8443468-a026-4333-b112-87ba65b465ff None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:39.251690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8443468-a026-4333-b112-87ba65b465ff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:39.253632 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8443468-a026-4333-b112-87ba65b465ff None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:39.264145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 403/800] 10.222.0.47 () {56 vars in 1124 bytes} [Sun Aug 21 01:05:39 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:39.264145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8443468-a026-4333-b112-87ba65b465ff None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:39.346497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:39.380297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de26b7e-180e-4659-ba32-79b0e859934a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:39.399270 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8443468-a026-4333-b112-87ba65b465ff None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:39.441892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db06b97b-af3f-4245-8a30-4302a3a3a46c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:39.443511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db06b97b-af3f-4245-8a30-4302a3a3a46c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:39.444369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db06b97b-af3f-4245-8a30-4302a3a3a46c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:39.445120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db06b97b-af3f-4245-8a30-4302a3a3a46c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:39.445405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8766a8-bb60-43d4-ae3e-a6c585bab75d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:39.446132 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8766a8-bb60-43d4-ae3e-a6c585bab75d None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:39.446410 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8766a8-bb60-43d4-ae3e-a6c585bab75d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:39.446681 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8766a8-bb60-43d4-ae3e-a6c585bab75d None admin] PATH_INFO: `/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/06a5972a55824711b06670b925322739/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:39.448754 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8766a8-bb60-43d4-ae3e-a6c585bab75d None admin] RBAC: Authorizing `identity:create_grant(project_id=75a3bf3f3a4244d18fde47d492657db4, user_id=06a5972a55824711b06670b925322739, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:39.450506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8766a8-bb60-43d4-ae3e-a6c585bab75d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:39.463195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db06b97b-af3f-4245-8a30-4302a3a3a46c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:39.464670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db06b97b-af3f-4245-8a30-4302a3a3a46c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:39.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-db06b97b-af3f-4245-8a30-4302a3a3a46c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:39.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:39.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:39.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:39.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:39.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:39.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:39.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:39.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:39.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:39.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:39.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:39.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:39.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:39.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:39.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:39.480796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:39.482408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:39.482408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:39.482408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:39.482408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 398/801] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 452 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:39.482408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db06b97b-af3f-4245-8a30-4302a3a3a46c None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:39.550898 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 404/802] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:05:39 2022] PUT /identity/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/06a5972a55824711b06670b925322739/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:05:39.551727 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8766a8-bb60-43d4-ae3e-a6c585bab75d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:39.608147 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:39.660494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db06b97b-af3f-4245-8a30-4302a3a3a46c None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:39.700432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff281a38-9235-4097-90de-ef85c0b64f90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:39.701355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff281a38-9235-4097-90de-ef85c0b64f90 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:39.701355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff281a38-9235-4097-90de-ef85c0b64f90 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:39.701814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff281a38-9235-4097-90de-ef85c0b64f90 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:39.702956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff281a38-9235-4097-90de-ef85c0b64f90 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:39.702956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff281a38-9235-4097-90de-ef85c0b64f90 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:39.704144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff281a38-9235-4097-90de-ef85c0b64f90 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:39.709905 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 399/803] 10.222.0.47 () {56 vars in 1124 bytes} [Sun Aug 21 01:05:39 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:39.711251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9678d3c6-8b52-485a-bc6b-d1aa422d8d42 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:39.711475 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9678d3c6-8b52-485a-bc6b-d1aa422d8d42 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:39.711745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9678d3c6-8b52-485a-bc6b-d1aa422d8d42 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:39.713417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 400/804] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:39.716555 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9678d3c6-8b52-485a-bc6b-d1aa422d8d42 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:39.751039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8766a8-bb60-43d4-ae3e-a6c585bab75d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:39.773725 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:39.799033 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d339c0-7b8c-4df0-8178-b1916d6f52e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:39.800260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d339c0-7b8c-4df0-8178-b1916d6f52e4 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:39.800541 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d339c0-7b8c-4df0-8178-b1916d6f52e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:39.800809 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d339c0-7b8c-4df0-8178-b1916d6f52e4 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:39.801541 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d339c0-7b8c-4df0-8178-b1916d6f52e4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:39.801977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d339c0-7b8c-4df0-8178-b1916d6f52e4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:39.804551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d339c0-7b8c-4df0-8178-b1916d6f52e4 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:39.821728 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 405/805] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:39.824060 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5010f2e3-22da-4b65-9900-67c9c52ecf7a None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:39.824378 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5010f2e3-22da-4b65-9900-67c9c52ecf7a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:39.824657 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5010f2e3-22da-4b65-9900-67c9c52ecf7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:39.849679 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-5010f2e3-22da-4b65-9900-67c9c52ecf7a None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:39.851293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5010f2e3-22da-4b65-9900-67c9c52ecf7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:39.877423 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9678d3c6-8b52-485a-bc6b-d1aa422d8d42 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:39.899830 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:39.915761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e4aa65-78b2-4308-b5f1-283263f2c051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:39.916429 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e4aa65-78b2-4308-b5f1-283263f2c051 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:39.916745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e4aa65-78b2-4308-b5f1-283263f2c051 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:39.917097 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e4aa65-78b2-4308-b5f1-283263f2c051 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:39.917793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e4aa65-78b2-4308-b5f1-283263f2c051 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:39.919165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e4aa65-78b2-4308-b5f1-283263f2c051 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:39.940131 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 406/806] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:39 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:39.940131 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5010f2e3-22da-4b65-9900-67c9c52ecf7a None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:40.033047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 401/807] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:05:39 2022] POST /identity/v3/endpoints => generated 346 bytes in 317 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:05:40.037070 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fec0b3-0751-4fbf-b57c-f8e7e8cd814b None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:40.038691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fec0b3-0751-4fbf-b57c-f8e7e8cd814b None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:40.039358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fec0b3-0751-4fbf-b57c-f8e7e8cd814b None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:40.056301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 402/808] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:40 2022] GET /identity => generated 270 bytes in 21 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:40.062413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707f0366-207c-4eaa-8ac2-ad1a312e32c7 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:40.062692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707f0366-207c-4eaa-8ac2-ad1a312e32c7 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:40.062975 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707f0366-207c-4eaa-8ac2-ad1a312e32c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:40.064807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5010f2e3-22da-4b65-9900-67c9c52ecf7a None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:40.078962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:40.112990 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-707f0366-207c-4eaa-8ac2-ad1a312e32c7 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:40.114382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-707f0366-207c-4eaa-8ac2-ad1a312e32c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:40.148874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00aaa8ab-1d28-4030-8d52-60998b0efa98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:40.149436 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00aaa8ab-1d28-4030-8d52-60998b0efa98 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:40.149710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00aaa8ab-1d28-4030-8d52-60998b0efa98 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:40.149978 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00aaa8ab-1d28-4030-8d52-60998b0efa98 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:40.164263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00aaa8ab-1d28-4030-8d52-60998b0efa98 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:40.165810 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00aaa8ab-1d28-4030-8d52-60998b0efa98 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:40.207078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-00aaa8ab-1d28-4030-8d52-60998b0efa98 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:05:40.207078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:40.207078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:40.207078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:40.207078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:40.207078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:40.207078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:40.207078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:40.207078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:40.207078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:40.207078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:40.207078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:40.207078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:40.207078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:40.207078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:40.207078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:40.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:40.209122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:40.209122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:40.209122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:40.209122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:40.209122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:40.209122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:40.209122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:40.209122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:40.209122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:05:40.209122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:40.209122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 407/809] 10.222.0.47 () {56 vars in 1093 bytes} [Sun Aug 21 01:05:39 2022] GET /identity/v3/users/nova => generated 82 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:40.214779 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00aaa8ab-1d28-4030-8d52-60998b0efa98 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:40.267597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 403/810] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:40 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:40.268537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707f0366-207c-4eaa-8ac2-ad1a312e32c7 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:40.355892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00aaa8ab-1d28-4030-8d52-60998b0efa98 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:40.393732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94ac7ae-1b49-4a97-b3f3-b9c91586bef1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:40.394345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94ac7ae-1b49-4a97-b3f3-b9c91586bef1 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:40.394617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94ac7ae-1b49-4a97-b3f3-b9c91586bef1 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:40.394882 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94ac7ae-1b49-4a97-b3f3-b9c91586bef1 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:40.395603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94ac7ae-1b49-4a97-b3f3-b9c91586bef1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:40.395900 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94ac7ae-1b49-4a97-b3f3-b9c91586bef1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:40.397359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94ac7ae-1b49-4a97-b3f3-b9c91586bef1 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:40.402803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707f0366-207c-4eaa-8ac2-ad1a312e32c7 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:40.411862 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:40.416574 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 408/811] 10.222.0.47 () {56 vars in 1133 bytes} [Sun Aug 21 01:05:40 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:40.422753 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94ac7ae-1b49-4a97-b3f3-b9c91586bef1 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:40.446940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce1c00b-d266-4fcf-98f0-5705192b879a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:40.447582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce1c00b-d266-4fcf-98f0-5705192b879a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:40.447846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce1c00b-d266-4fcf-98f0-5705192b879a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:40.448249 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce1c00b-d266-4fcf-98f0-5705192b879a None admin] PATH_INFO: `/v3/services/network` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:40.449567 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce1c00b-d266-4fcf-98f0-5705192b879a None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:40.451385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce1c00b-d266-4fcf-98f0-5705192b879a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-8ce1c00b-d266-4fcf-98f0-5705192b879a None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:40.462817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:40.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:40.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:40.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:40.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:40.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:40.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:40.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:05:40.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:05:40.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:05:40.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:05:40.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:40.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:40.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:40.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:40.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:40.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:40.465537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:40.465537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:40.465537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:40.465537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:40.465537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:40.465537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:40.465537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:40.465537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:40.465537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:40.465537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:40.465537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:05:40.465537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:40.465537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 21 01:05:40.465537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:40.465537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 404/812] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:40 2022] GET /identity/v3/services/network => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:40.474886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce1c00b-d266-4fcf-98f0-5705192b879a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:40.478952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:40.625293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94ac7ae-1b49-4a97-b3f3-b9c91586bef1 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:40.661029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce1c00b-d266-4fcf-98f0-5705192b879a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:40.666777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1f2cd7-bdfd-4ae4-a7a1-dde84891a18e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:40.667389 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1f2cd7-bdfd-4ae4-a7a1-dde84891a18e None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:40.667662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1f2cd7-bdfd-4ae4-a7a1-dde84891a18e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:40.667930 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1f2cd7-bdfd-4ae4-a7a1-dde84891a18e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:40.702290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f15e49-7494-43c6-a14b-cda664cf7b6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:40.703564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f15e49-7494-43c6-a14b-cda664cf7b6a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:40.703849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f15e49-7494-43c6-a14b-cda664cf7b6a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:40.704141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f15e49-7494-43c6-a14b-cda664cf7b6a None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:40.704785 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f15e49-7494-43c6-a14b-cda664cf7b6a None admin] RBAC: Adding query filter params (name=network) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:40.705139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f15e49-7494-43c6-a14b-cda664cf7b6a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:40.708346 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f15e49-7494-43c6-a14b-cda664cf7b6a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:40.714158 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 405/813] 10.222.0.47 () {56 vars in 1112 bytes} [Sun Aug 21 01:05:40 2022] GET /identity/v3/services?name=network => generated 124 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:40.715494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f15e49-7494-43c6-a14b-cda664cf7b6a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:40.747877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1f2cd7-bdfd-4ae4-a7a1-dde84891a18e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:40.749486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1f2cd7-bdfd-4ae4-a7a1-dde84891a18e None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:40.760386 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:40.782133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:40.834358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f15e49-7494-43c6-a14b-cda664cf7b6a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:40.867205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 409/814] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:40 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:05:40.869108 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1f2cd7-bdfd-4ae4-a7a1-dde84891a18e None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:40.872148 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4f2795-a7f5-4418-9bc9-d0d42af071b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:40.872800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4f2795-a7f5-4418-9bc9-d0d42af071b2 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:40.873114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4f2795-a7f5-4418-9bc9-d0d42af071b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:40.873348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4f2795-a7f5-4418-9bc9-d0d42af071b2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:40.885050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4f2795-a7f5-4418-9bc9-d0d42af071b2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:40.890195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-ec4f2795-a7f5-4418-9bc9-d0d42af071b2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:40.890195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:40.890195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:40.890195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:40.890195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:40.890195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:40.890195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:40.890195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:40.890195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:40.890195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:40.890195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:40.890195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:40.890195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:40.890195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:40.890195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:40.890195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:40.892412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:40.892412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:40.892412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:40.892412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:40.892412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:40.892412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:40.892412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:40.892412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:40.892412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:40.892412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:40.892412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:40.892412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:40.892412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 406/815] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:40 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:40.892412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4f2795-a7f5-4418-9bc9-d0d42af071b2 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:40.998106 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1f2cd7-bdfd-4ae4-a7a1-dde84891a18e None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:41.021540 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:41.023270 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:41.032136 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4f2795-a7f5-4418-9bc9-d0d42af071b2 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:41.036254 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfae8c7c-a344-43a0-9e5d-0da548948d61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:41.036936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfae8c7c-a344-43a0-9e5d-0da548948d61 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:41.037284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfae8c7c-a344-43a0-9e5d-0da548948d61 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:41.037626 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfae8c7c-a344-43a0-9e5d-0da548948d61 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:41.038322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfae8c7c-a344-43a0-9e5d-0da548948d61 None admin] RBAC: Adding query filter params (type=network) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:41.038691 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfae8c7c-a344-43a0-9e5d-0da548948d61 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:41.040124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfae8c7c-a344-43a0-9e5d-0da548948d61 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:41.045046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 410/816] 10.222.0.47 () {56 vars in 1112 bytes} [Sun Aug 21 01:05:40 2022] GET /identity/v3/services?type=network => generated 350 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:41.045907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfae8c7c-a344-43a0-9e5d-0da548948d61 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:41.078871 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e51d2e-6e79-460a-84bd-86a37e3cce94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:41.079847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e51d2e-6e79-460a-84bd-86a37e3cce94 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:41.080332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e51d2e-6e79-460a-84bd-86a37e3cce94 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:41.080746 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e51d2e-6e79-460a-84bd-86a37e3cce94 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:41.095997 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e51d2e-6e79-460a-84bd-86a37e3cce94 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:41.097957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e51d2e-6e79-460a-84bd-86a37e3cce94 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:41.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-d9e51d2e-6e79-460a-84bd-86a37e3cce94 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:41.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:41.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:41.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:41.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:41.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:41.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:41.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:41.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:41.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:41.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:41.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:41.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:41.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:41.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:41.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:41.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:41.119494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:41.119494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:41.119494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:41.121898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 407/817] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:41.122849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e51d2e-6e79-460a-84bd-86a37e3cce94 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:41.161768 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfae8c7c-a344-43a0-9e5d-0da548948d61 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:41.198876 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc46adc-d4d9-4025-a150-dac50c39742d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:41.199630 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc46adc-d4d9-4025-a150-dac50c39742d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:41.200003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc46adc-d4d9-4025-a150-dac50c39742d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:41.200400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc46adc-d4d9-4025-a150-dac50c39742d None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:41.201181 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc46adc-d4d9-4025-a150-dac50c39742d None admin] RBAC: Adding query filter params (name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:41.201537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc46adc-d4d9-4025-a150-dac50c39742d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:41.202977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc46adc-d4d9-4025-a150-dac50c39742d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:41.216365 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 411/818] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:41.217459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc46adc-d4d9-4025-a150-dac50c39742d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:41.260687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e51d2e-6e79-460a-84bd-86a37e3cce94 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:41.271515 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:41.301699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e877966e-efd8-4473-bd14-ee2aa7dd1cf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:41.302743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e877966e-efd8-4473-bd14-ee2aa7dd1cf9 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:41.303194 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e877966e-efd8-4473-bd14-ee2aa7dd1cf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:41.303747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e877966e-efd8-4473-bd14-ee2aa7dd1cf9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:41.305058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e877966e-efd8-4473-bd14-ee2aa7dd1cf9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:41.307034 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e877966e-efd8-4473-bd14-ee2aa7dd1cf9 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:41.331174 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc46adc-d4d9-4025-a150-dac50c39742d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:41.372728 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6237aabf-6273-4245-b7cf-c83de20b74f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:41.373352 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6237aabf-6273-4245-b7cf-c83de20b74f5 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:41.373584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6237aabf-6273-4245-b7cf-c83de20b74f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:41.373858 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6237aabf-6273-4245-b7cf-c83de20b74f5 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:41.374495 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6237aabf-6273-4245-b7cf-c83de20b74f5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:41.374797 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6237aabf-6273-4245-b7cf-c83de20b74f5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:41.376184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6237aabf-6273-4245-b7cf-c83de20b74f5 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:41.393153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 412/819] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:41.393153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6237aabf-6273-4245-b7cf-c83de20b74f5 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:41.401831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 408/820] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:05:41 2022] POST /identity/v3/endpoints => generated 337 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:05:41.402515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e877966e-efd8-4473-bd14-ee2aa7dd1cf9 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:41.406859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:41.555798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e877966e-efd8-4473-bd14-ee2aa7dd1cf9 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:41.599301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c24d0c7-a07f-4852-ac4e-9e632ddb3cca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:41.600348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c24d0c7-a07f-4852-ac4e-9e632ddb3cca None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:41.601437 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c24d0c7-a07f-4852-ac4e-9e632ddb3cca None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:41.602048 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c24d0c7-a07f-4852-ac4e-9e632ddb3cca None admin] PATH_INFO: `/v3/projects/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:41.611857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c24d0c7-a07f-4852-ac4e-9e632ddb3cca None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:41.613836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c24d0c7-a07f-4852-ac4e-9e632ddb3cca None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:41.614478 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6237aabf-6273-4245-b7cf-c83de20b74f5 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:41.629184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-6c24d0c7-a07f-4852-ac4e-9e632ddb3cca None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:41.629184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:41.629184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:41.629184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:41.629184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:41.629184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:41.629184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:41.629184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:41.629184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:41.629184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:41.629184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:41.629184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:41.629184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:41.629184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:41.629184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:41.630157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:41.631065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:41.631065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:41.631065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:41.631065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:41.631065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:41.631065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:41.631065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:41.631065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:41.633728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 409/821] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:41.640247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:41.642081 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c24d0c7-a07f-4852-ac4e-9e632ddb3cca None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:41.675286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8742d695-5d71-4f63-851e-f794572027f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:41.675936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8742d695-5d71-4f63-851e-f794572027f8 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:41.676244 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8742d695-5d71-4f63-851e-f794572027f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:41.676585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8742d695-5d71-4f63-851e-f794572027f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:41.691207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8742d695-5d71-4f63-851e-f794572027f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:41.692794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8742d695-5d71-4f63-851e-f794572027f8 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:41.735736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:41.739766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-8742d695-5d71-4f63-851e-f794572027f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:41.739766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:41.739766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:41.739766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:41.739766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:41.739766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:41.739766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:41.739766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:41.739766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:41.739766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:41.739766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:41.739766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:41.739766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:41.739766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:41.739766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:41.739766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:41.741585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:41.744310 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:41.744310 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:41.744310 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:41.747233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 413/822] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 358 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:41.754869 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8742d695-5d71-4f63-851e-f794572027f8 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:41.818614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c24d0c7-a07f-4852-ac4e-9e632ddb3cca None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:41.873964 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab6f5e3-87a0-45ec-9a1c-42d87a8ce64f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:41.874843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab6f5e3-87a0-45ec-9a1c-42d87a8ce64f None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:41.875119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab6f5e3-87a0-45ec-9a1c-42d87a8ce64f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:41.875384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab6f5e3-87a0-45ec-9a1c-42d87a8ce64f None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:41.876069 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab6f5e3-87a0-45ec-9a1c-42d87a8ce64f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:41.876391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab6f5e3-87a0-45ec-9a1c-42d87a8ce64f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:41.878577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab6f5e3-87a0-45ec-9a1c-42d87a8ce64f None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:41.892614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 410/823] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:41.895105 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c217459e-c879-4154-a1d3-5affcb3cb905 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:41.895629 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c217459e-c879-4154-a1d3-5affcb3cb905 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:41.896348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c217459e-c879-4154-a1d3-5affcb3cb905 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:41.898576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 411/824] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:41 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:41.899772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c217459e-c879-4154-a1d3-5affcb3cb905 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:41.906363 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8742d695-5d71-4f63-851e-f794572027f8 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:41.958421 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e1c924-3a7a-40c4-8a19-427ce8e0391b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:41.959276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e1c924-3a7a-40c4-8a19-427ce8e0391b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:41.959438 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e1c924-3a7a-40c4-8a19-427ce8e0391b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:41.959785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e1c924-3a7a-40c4-8a19-427ce8e0391b None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:41.960369 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e1c924-3a7a-40c4-8a19-427ce8e0391b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:41.960677 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e1c924-3a7a-40c4-8a19-427ce8e0391b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:41.962035 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e1c924-3a7a-40c4-8a19-427ce8e0391b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:41.975468 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 414/825] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:41.976841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905b7570-0141-4470-bda4-d3c2e1c2a879 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:41.977127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905b7570-0141-4470-bda4-d3c2e1c2a879 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:41.977399 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905b7570-0141-4470-bda4-d3c2e1c2a879 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:41.996397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-905b7570-0141-4470-bda4-d3c2e1c2a879 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:41.998111 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-905b7570-0141-4470-bda4-d3c2e1c2a879 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:42.014536 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:42.023193 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c217459e-c879-4154-a1d3-5affcb3cb905 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:42.042076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:42.060564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f7330e-dd3c-4fac-a5dd-07613317be7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:42.061443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f7330e-dd3c-4fac-a5dd-07613317be7a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:42.061722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f7330e-dd3c-4fac-a5dd-07613317be7a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:42.061997 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f7330e-dd3c-4fac-a5dd-07613317be7a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:42.062795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f7330e-dd3c-4fac-a5dd-07613317be7a None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=fabdcdc25d1d4ef6bc66bd6cadecaaf8) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:42.063102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f7330e-dd3c-4fac-a5dd-07613317be7a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:42.064909 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f7330e-dd3c-4fac-a5dd-07613317be7a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:42.071366 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 412/826] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:41 2022] GET /identity/v3/role_assignments?user.id=fabdcdc25d1d4ef6bc66bd6cadecaaf8&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:42.072121 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f7330e-dd3c-4fac-a5dd-07613317be7a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:42.086429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 415/827] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:41 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:42.200095 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f7330e-dd3c-4fac-a5dd-07613317be7a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:42.244096 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e84b9a-e4c9-4062-bd30-2353326c5dee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:42.245168 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e84b9a-e4c9-4062-bd30-2353326c5dee None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:42.245624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e84b9a-e4c9-4062-bd30-2353326c5dee None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:42.246068 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e84b9a-e4c9-4062-bd30-2353326c5dee None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:42.256877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e84b9a-e4c9-4062-bd30-2353326c5dee None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:42.258601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e84b9a-e4c9-4062-bd30-2353326c5dee None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:42.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-65e84b9a-e4c9-4062-bd30-2353326c5dee None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 21 01:05:42.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:42.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:42.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:42.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:42.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:42.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:42.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:42.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:42.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:42.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:42.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:42.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:42.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:42.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:42.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:42.275327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:42.276370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:42.276370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:42.276370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:42.276370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:42.276370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:42.276370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:42.276370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:42.276370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:42.276370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 21 01:05:42.276370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:42.276370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 413/828] 10.222.0.47 () {56 vars in 1117 bytes} [Sun Aug 21 01:05:42 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:42.281960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905b7570-0141-4470-bda4-d3c2e1c2a879 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:42.305219 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b3323b-2bbc-46db-a3f2-f36e6feeea29 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:42.305557 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b3323b-2bbc-46db-a3f2-f36e6feeea29 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:42.305839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b3323b-2bbc-46db-a3f2-f36e6feeea29 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:42.308339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 414/829] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:42.315436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8321726-b082-48df-ae5d-1aa78b7a1743 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:42.315806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8321726-b082-48df-ae5d-1aa78b7a1743 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:42.316345 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8321726-b082-48df-ae5d-1aa78b7a1743 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:42.335476 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-a8321726-b082-48df-ae5d-1aa78b7a1743 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:42.337014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-a8321726-b082-48df-ae5d-1aa78b7a1743 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:42.385489 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:42.412295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905b7570-0141-4470-bda4-d3c2e1c2a879 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:42.424444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 415/830] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:42 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:05:42.425607 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8321726-b082-48df-ae5d-1aa78b7a1743 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:42.457605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e4d7c5-1ed4-4454-b887-830169716145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:42.458635 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4d7c5-1ed4-4454-b887-830169716145 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:42.459247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4d7c5-1ed4-4454-b887-830169716145 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:42.459843 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4d7c5-1ed4-4454-b887-830169716145 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:42.460862 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e4d7c5-1ed4-4454-b887-830169716145 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:42.461429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e4d7c5-1ed4-4454-b887-830169716145 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:42.463183 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e4d7c5-1ed4-4454-b887-830169716145 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:42.488331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 416/831] 10.222.0.47 () {56 vars in 1149 bytes} [Sun Aug 21 01:05:42 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:42.490383 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e4d7c5-1ed4-4454-b887-830169716145 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:42.504652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:42.539018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8321726-b082-48df-ae5d-1aa78b7a1743 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:42.580218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554a7574-82da-4b89-8b1c-9ccf0fe16f59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:42.580829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554a7574-82da-4b89-8b1c-9ccf0fe16f59 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:42.581104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554a7574-82da-4b89-8b1c-9ccf0fe16f59 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:42.581374 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554a7574-82da-4b89-8b1c-9ccf0fe16f59 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:42.593328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554a7574-82da-4b89-8b1c-9ccf0fe16f59 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:42.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-554a7574-82da-4b89-8b1c-9ccf0fe16f59 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:05:42.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:42.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:42.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:42.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:42.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:42.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:42.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:42.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:42.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:42.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:42.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:42.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:42.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:42.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:42.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:42.599336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:42.599336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:42.599336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:42.599336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:42.599336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:42.599336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:42.599336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:42.599336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:42.599336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:42.599336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:42.599336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:42.599336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:42.599336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:42.599336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:42.599336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:42.599336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:42.600258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:42.600258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:42.600258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:42.600258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:42.600258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:42.600258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:42.600258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:42.600258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:42.600258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:42.600258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:42.600258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:05:42.600258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:42.604401 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 416/832] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:05:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:42.605515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554a7574-82da-4b89-8b1c-9ccf0fe16f59 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:42.622898 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e4d7c5-1ed4-4454-b887-830169716145 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:42.641489 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:42.664322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea676c8-6002-4d32-b3d3-a477c8d7f765 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:42.665428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea676c8-6002-4d32-b3d3-a477c8d7f765 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:42.665925 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea676c8-6002-4d32-b3d3-a477c8d7f765 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:42.666398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea676c8-6002-4d32-b3d3-a477c8d7f765 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:42.679741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea676c8-6002-4d32-b3d3-a477c8d7f765 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:42.681861 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea676c8-6002-4d32-b3d3-a477c8d7f765 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:42.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-eea676c8-6002-4d32-b3d3-a477c8d7f765 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:42.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:42.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:42.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:42.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:42.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:42.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:42.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:42.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:42.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:42.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:42.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:42.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:42.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:42.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:42.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:42.700224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:42.701144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:42.701144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:42.701144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:42.708176 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 417/833] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:42.709597 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea676c8-6002-4d32-b3d3-a477c8d7f765 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:42.717967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554a7574-82da-4b89-8b1c-9ccf0fe16f59 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:42.755450 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b848568f-446e-432b-a341-b0cf37834ce5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:42.758058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b848568f-446e-432b-a341-b0cf37834ce5 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:42.758058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b848568f-446e-432b-a341-b0cf37834ce5 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:42.758058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b848568f-446e-432b-a341-b0cf37834ce5 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:42.758058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b848568f-446e-432b-a341-b0cf37834ce5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:42.758058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b848568f-446e-432b-a341-b0cf37834ce5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:42.758877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b848568f-446e-432b-a341-b0cf37834ce5 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:42.770045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 417/834] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:05:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:42.770934 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b848568f-446e-432b-a341-b0cf37834ce5 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:42.785375 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:42.837102 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea676c8-6002-4d32-b3d3-a477c8d7f765 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:42.873628 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:42.875066 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82627b29-da3f-40a6-a7e6-41ae47f7d4c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:42.875755 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82627b29-da3f-40a6-a7e6-41ae47f7d4c8 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:42.876091 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82627b29-da3f-40a6-a7e6-41ae47f7d4c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:42.876383 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82627b29-da3f-40a6-a7e6-41ae47f7d4c8 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:42.877060 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82627b29-da3f-40a6-a7e6-41ae47f7d4c8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:42.877359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82627b29-da3f-40a6-a7e6-41ae47f7d4c8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:42.878704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82627b29-da3f-40a6-a7e6-41ae47f7d4c8 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:42.881187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b848568f-446e-432b-a341-b0cf37834ce5 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:42.895778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 418/835] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:42.895778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82627b29-da3f-40a6-a7e6-41ae47f7d4c8 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:42.930698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398e585-8b59-42aa-8d69-4ef7b6797f1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:42.931334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398e585-8b59-42aa-8d69-4ef7b6797f1d None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:42.931610 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398e585-8b59-42aa-8d69-4ef7b6797f1d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:42.931880 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398e585-8b59-42aa-8d69-4ef7b6797f1d None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:42.932493 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c398e585-8b59-42aa-8d69-4ef7b6797f1d None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:42.933852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c398e585-8b59-42aa-8d69-4ef7b6797f1d None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-c398e585-8b59-42aa-8d69-4ef7b6797f1d None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:42.943102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:42.945056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:42.945056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:42.945056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:42.945056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:42.945056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:42.945056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:42.945056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:05:42.945056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:05:42.945056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:05:42.945056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:05:42.945056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:42.945056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:42.945056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:42.945056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:42.945056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:42.945056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:42.947061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:42.947061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:42.947061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:42.947061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:42.947061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:42.947061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:42.947061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:42.947061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:42.947061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:42.947061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:42.947061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:05:42.947061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:42.947061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 21 01:05:42.947061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:42.947061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 418/836] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:05:42 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:42.949542 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398e585-8b59-42aa-8d69-4ef7b6797f1d None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:43.019528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82627b29-da3f-40a6-a7e6-41ae47f7d4c8 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:43.041738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:43.056727 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6b8044-8755-4ce9-bdf2-de168a49c7a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:43.057690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6b8044-8755-4ce9-bdf2-de168a49c7a1 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:43.057983 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6b8044-8755-4ce9-bdf2-de168a49c7a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:43.058241 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6b8044-8755-4ce9-bdf2-de168a49c7a1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:43.064400 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398e585-8b59-42aa-8d69-4ef7b6797f1d None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:43.069067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6b8044-8755-4ce9-bdf2-de168a49c7a1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:43.070663 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6b8044-8755-4ce9-bdf2-de168a49c7a1 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:43.087413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-1b6b8044-8755-4ce9-bdf2-de168a49c7a1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:43.087413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:43.087413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:43.087413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:43.087413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:43.087413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:43.087413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:43.087413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:43.087413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:43.087413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:43.087413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:43.087413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:43.087413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:43.087413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:43.087413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:43.087413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:43.088903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:43.089837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:43.089837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:43.089837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:43.089837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 419/837] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:43.089837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6b8044-8755-4ce9-bdf2-de168a49c7a1 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:43.107448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51cef9c8-d729-4aca-829e-14d823365d17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:43.108082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cef9c8-d729-4aca-829e-14d823365d17 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:43.108391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cef9c8-d729-4aca-829e-14d823365d17 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:43.108662 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cef9c8-d729-4aca-829e-14d823365d17 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:43.115730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51cef9c8-d729-4aca-829e-14d823365d17 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:43.117214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51cef9c8-d729-4aca-829e-14d823365d17 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:43.128961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-51cef9c8-d729-4aca-829e-14d823365d17 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:43.128961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:43.128961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:43.128961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:43.128961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:43.128961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:43.128961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:43.128961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:43.128961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:43.128961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:43.128961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:43.128961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:43.128961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:43.128961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:43.128961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:43.130843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:43.131927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:43.131927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:43.131927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:43.131927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:43.131927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:43.131927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:43.131927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:43.131927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:43.131927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 419/838] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:43.131927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51cef9c8-d729-4aca-829e-14d823365d17 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:43.155560 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:43.224422 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6b8044-8755-4ce9-bdf2-de168a49c7a1 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:43.262103 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8b4260-5c0e-42e6-b071-208511df2849 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:43.263056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8b4260-5c0e-42e6-b071-208511df2849 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:43.263520 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8b4260-5c0e-42e6-b071-208511df2849 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:43.263967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8b4260-5c0e-42e6-b071-208511df2849 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:43.264812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8b4260-5c0e-42e6-b071-208511df2849 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:43.265113 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8b4260-5c0e-42e6-b071-208511df2849 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:43.266461 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8b4260-5c0e-42e6-b071-208511df2849 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:43.271725 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 420/839] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:05:43 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:43.272542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8b4260-5c0e-42e6-b071-208511df2849 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:43.278750 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51cef9c8-d729-4aca-829e-14d823365d17 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:43.313426 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:43.316410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52afaf28-604d-40d6-8800-f2a060f612ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:43.317054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52afaf28-604d-40d6-8800-f2a060f612ce None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:43.317425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52afaf28-604d-40d6-8800-f2a060f612ce None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:43.317864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52afaf28-604d-40d6-8800-f2a060f612ce None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:43.318225 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52afaf28-604d-40d6-8800-f2a060f612ce None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:43.318505 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52afaf28-604d-40d6-8800-f2a060f612ce None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:43.319912 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52afaf28-604d-40d6-8800-f2a060f612ce None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:43.337929 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 420/840] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:43.337929 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52afaf28-604d-40d6-8800-f2a060f612ce None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:43.402324 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8b4260-5c0e-42e6-b071-208511df2849 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:43.428979 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:43.443357 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfc774a-7f36-49b0-8288-9985899a7c7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:43.444067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfc774a-7f36-49b0-8288-9985899a7c7d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:43.444374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfc774a-7f36-49b0-8288-9985899a7c7d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:43.444649 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfc774a-7f36-49b0-8288-9985899a7c7d None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:43.445342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfc774a-7f36-49b0-8288-9985899a7c7d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:43.445645 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfc774a-7f36-49b0-8288-9985899a7c7d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:43.447070 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfc774a-7f36-49b0-8288-9985899a7c7d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:43.452953 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52afaf28-604d-40d6-8800-f2a060f612ce None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:43.464524 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 421/841] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:43.464524 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfc774a-7f36-49b0-8288-9985899a7c7d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:43.495131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa13e604-4722-46c7-8dbf-db037f8b9303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:43.495737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa13e604-4722-46c7-8dbf-db037f8b9303 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:43.496033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa13e604-4722-46c7-8dbf-db037f8b9303 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:43.496308 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa13e604-4722-46c7-8dbf-db037f8b9303 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:43.496932 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa13e604-4722-46c7-8dbf-db037f8b9303 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:43.497227 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa13e604-4722-46c7-8dbf-db037f8b9303 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:43.498573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa13e604-4722-46c7-8dbf-db037f8b9303 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:43.505030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 421/842] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:05:43 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:43.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa13e604-4722-46c7-8dbf-db037f8b9303 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:43.599225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:43.606267 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfc774a-7f36-49b0-8288-9985899a7c7d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:43.647741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525302fb-8f62-4c2e-bcdc-4a40dcba5950 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:43.648601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525302fb-8f62-4c2e-bcdc-4a40dcba5950 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:43.649062 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525302fb-8f62-4c2e-bcdc-4a40dcba5950 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:43.649451 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525302fb-8f62-4c2e-bcdc-4a40dcba5950 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:43.652783 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa13e604-4722-46c7-8dbf-db037f8b9303 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:43.658568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525302fb-8f62-4c2e-bcdc-4a40dcba5950 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:43.660209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525302fb-8f62-4c2e-bcdc-4a40dcba5950 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:43.677051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-525302fb-8f62-4c2e-bcdc-4a40dcba5950 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 21 01:05:43.677051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:43.677051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:43.677051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:43.677051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:43.677051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:43.677051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:43.677051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:43.677051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:43.677051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:43.677051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:43.677051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:43.677051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:43.677051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:43.677051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:43.677051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:43.681824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:43.681824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:43.681824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:43.681824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:43.681824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:43.681824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:43.681824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:43.681824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:43.681824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 21 01:05:43.681824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:43.681824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 422/843] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:05:43 2022] GET /identity/v3/users/swift => generated 83 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:43.685968 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525302fb-8f62-4c2e-bcdc-4a40dcba5950 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:43.714533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c29bdd-31d2-4bbd-acb9-36815fd4f9a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:43.714783 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c29bdd-31d2-4bbd-acb9-36815fd4f9a2 None admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:43.715035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c29bdd-31d2-4bbd-acb9-36815fd4f9a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:43.715307 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c29bdd-31d2-4bbd-acb9-36815fd4f9a2 None admin] PATH_INFO: `/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/d7c65195ff8d42958fa54b5f083bf8da/roles/c2d8bf7283e14857879d03d1f39c0165` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:43.740658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:43.752869 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c29bdd-31d2-4bbd-acb9-36815fd4f9a2 None admin] RBAC: Authorizing `identity:create_grant(project_id=75a3bf3f3a4244d18fde47d492657db4, user_id=d7c65195ff8d42958fa54b5f083bf8da, role_id=c2d8bf7283e14857879d03d1f39c0165)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:43.752869 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c29bdd-31d2-4bbd-acb9-36815fd4f9a2 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:43.823282 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525302fb-8f62-4c2e-bcdc-4a40dcba5950 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:43.857834 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354e2cf2-d583-456a-95d5-ccac2979e149 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:43.858438 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354e2cf2-d583-456a-95d5-ccac2979e149 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:43.858710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354e2cf2-d583-456a-95d5-ccac2979e149 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:43.858973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354e2cf2-d583-456a-95d5-ccac2979e149 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:43.859683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354e2cf2-d583-456a-95d5-ccac2979e149 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:43.859978 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354e2cf2-d583-456a-95d5-ccac2979e149 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:43.861946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354e2cf2-d583-456a-95d5-ccac2979e149 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:43.877276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 422/844] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:05:43 2022] PUT /identity/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/d7c65195ff8d42958fa54b5f083bf8da/roles/c2d8bf7283e14857879d03d1f39c0165 => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:05:43.888333 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 423/845] 10.222.0.47 () {56 vars in 1135 bytes} [Sun Aug 21 01:05:43 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:43.894906 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c29bdd-31d2-4bbd-acb9-36815fd4f9a2 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:44.011349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c29bdd-31d2-4bbd-acb9-36815fd4f9a2 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:44.049268 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e90b768-ffea-41c9-b094-331c2935786a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:44.049887 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e90b768-ffea-41c9-b094-331c2935786a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:44.050298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e90b768-ffea-41c9-b094-331c2935786a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:44.050569 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e90b768-ffea-41c9-b094-331c2935786a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:44.113022 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:44.122558 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e90b768-ffea-41c9-b094-331c2935786a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:44.124060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e90b768-ffea-41c9-b094-331c2935786a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:44.143206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed99c289-8e74-4c0d-b8dc-e21950ec16f1 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:44.143487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed99c289-8e74-4c0d-b8dc-e21950ec16f1 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:44.143762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed99c289-8e74-4c0d-b8dc-e21950ec16f1 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:44.145840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 424/846] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:44.153256 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997a0dd4-a2f8-4fe1-80cd-e6d080f82b6d None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:44.153665 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997a0dd4-a2f8-4fe1-80cd-e6d080f82b6d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:44.154061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997a0dd4-a2f8-4fe1-80cd-e6d080f82b6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:44.172828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-997a0dd4-a2f8-4fe1-80cd-e6d080f82b6d None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:44.174774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-997a0dd4-a2f8-4fe1-80cd-e6d080f82b6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:44.184887 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:44.234545 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 423/847] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:43 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:05:44.240704 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e90b768-ffea-41c9-b094-331c2935786a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:44.264453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 425/848] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:44 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:05:44.334812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:44.343473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e90b768-ffea-41c9-b094-331c2935786a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:44.380698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954bede8-c912-45c9-9a0e-978a1a30e245 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:44.381332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954bede8-c912-45c9-9a0e-978a1a30e245 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:44.381607 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954bede8-c912-45c9-9a0e-978a1a30e245 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:44.381876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954bede8-c912-45c9-9a0e-978a1a30e245 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:44.393228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954bede8-c912-45c9-9a0e-978a1a30e245 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:44.394660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954bede8-c912-45c9-9a0e-978a1a30e245 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:44.409150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-954bede8-c912-45c9-9a0e-978a1a30e245 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:44.409150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:44.409150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:44.409150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:44.409150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:44.409150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:44.409150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:44.409150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:44.409150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:44.409150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:44.409150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:44.409150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:44.409150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:44.409150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:44.409150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:44.409150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:44.410114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:44.411014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:44.411014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:44.411014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:44.411014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 424/849] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:44.417503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997a0dd4-a2f8-4fe1-80cd-e6d080f82b6d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:44.525538 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997a0dd4-a2f8-4fe1-80cd-e6d080f82b6d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:44.547730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954bede8-c912-45c9-9a0e-978a1a30e245 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:44.563090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35ffc82-eb8a-40de-8db9-0975a96797e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:44.564094 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35ffc82-eb8a-40de-8db9-0975a96797e2 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:44.565156 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35ffc82-eb8a-40de-8db9-0975a96797e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:44.565639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35ffc82-eb8a-40de-8db9-0975a96797e2 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:44.566768 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35ffc82-eb8a-40de-8db9-0975a96797e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:44.567290 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35ffc82-eb8a-40de-8db9-0975a96797e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:44.568891 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35ffc82-eb8a-40de-8db9-0975a96797e2 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:44.582109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 426/850] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:44.587758 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35ffc82-eb8a-40de-8db9-0975a96797e2 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:44.592195 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:44.654802 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954bede8-c912-45c9-9a0e-978a1a30e245 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:44.689631 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70de2b99-3967-4153-b71d-cf5f20b0e7bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:44.690281 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70de2b99-3967-4153-b71d-cf5f20b0e7bb None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:44.690556 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70de2b99-3967-4153-b71d-cf5f20b0e7bb None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:44.691076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70de2b99-3967-4153-b71d-cf5f20b0e7bb None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:44.691746 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70de2b99-3967-4153-b71d-cf5f20b0e7bb None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:44.693464 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70de2b99-3967-4153-b71d-cf5f20b0e7bb None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:44.700518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35ffc82-eb8a-40de-8db9-0975a96797e2 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:44.704405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-70de2b99-3967-4153-b71d-cf5f20b0e7bb None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 21 01:05:44.704405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:44.704405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:05:44.704405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:05:44.704405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:05:44.704405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:05:44.704405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:05:44.704405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:44.704405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 21 01:05:44.704405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:44.704405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:44.704405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:44.704405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:44.704405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:44.704405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:44.704405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:44.705879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:44.706782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:44.706782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:44.706782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:44.706782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:44.706782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:44.706782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:44.706782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:44.706782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:44.706782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:44.706782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:44.706782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:05:44.706782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:44.706782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 21 01:05:44.706782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:44.711093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 425/851] 10.222.0.47 () {56 vars in 1132 bytes} [Sun Aug 21 01:05:44 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:44.712087 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70de2b99-3967-4153-b71d-cf5f20b0e7bb None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:44.736571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:44.741464 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04ae098-a8d3-44ee-afe6-0b38d976bba5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:44.742396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04ae098-a8d3-44ee-afe6-0b38d976bba5 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:44.742823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04ae098-a8d3-44ee-afe6-0b38d976bba5 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:44.743239 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04ae098-a8d3-44ee-afe6-0b38d976bba5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:44.750602 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04ae098-a8d3-44ee-afe6-0b38d976bba5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:44.752422 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04ae098-a8d3-44ee-afe6-0b38d976bba5 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:44.765440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-f04ae098-a8d3-44ee-afe6-0b38d976bba5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:44.765440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:44.765440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:44.765440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:44.765440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:44.765440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:44.765440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:44.765440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:44.765440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:44.765440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:44.765440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:44.765440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:44.765440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:44.765440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:44.765440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:44.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:44.767936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:44.767936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:44.767936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:44.767936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:44.767936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:44.767936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:44.767936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:44.767936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:44.775514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 427/852] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:44.776757 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04ae098-a8d3-44ee-afe6-0b38d976bba5 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:44.821813 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70de2b99-3967-4153-b71d-cf5f20b0e7bb None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:44.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:44.859434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecccec1-a76a-4e6f-b07e-5bf9feb1b613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:44.861806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecccec1-a76a-4e6f-b07e-5bf9feb1b613 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:44.861806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecccec1-a76a-4e6f-b07e-5bf9feb1b613 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:44.861806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecccec1-a76a-4e6f-b07e-5bf9feb1b613 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:44.861806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecccec1-a76a-4e6f-b07e-5bf9feb1b613 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:44.862311 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecccec1-a76a-4e6f-b07e-5bf9feb1b613 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:44.863822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecccec1-a76a-4e6f-b07e-5bf9feb1b613 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:44.873312 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 426/853] 10.222.0.47 () {56 vars in 1126 bytes} [Sun Aug 21 01:05:44 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:44.874414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecccec1-a76a-4e6f-b07e-5bf9feb1b613 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:44.883860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04ae098-a8d3-44ee-afe6-0b38d976bba5 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:44.919699 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfaeb63-7997-4ac4-84cd-d11bd57dfd03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:44.920444 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfaeb63-7997-4ac4-84cd-d11bd57dfd03 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:44.920799 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfaeb63-7997-4ac4-84cd-d11bd57dfd03 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:44.921138 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfaeb63-7997-4ac4-84cd-d11bd57dfd03 None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:44.922116 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfaeb63-7997-4ac4-84cd-d11bd57dfd03 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:44.922226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfaeb63-7997-4ac4-84cd-d11bd57dfd03 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:44.923847 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfaeb63-7997-4ac4-84cd-d11bd57dfd03 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:44.943105 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 428/854] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:44.944214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfaeb63-7997-4ac4-84cd-d11bd57dfd03 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:44.964583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:44.987531 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecccec1-a76a-4e6f-b07e-5bf9feb1b613 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:45.026562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a22162e-d830-4718-ad92-411018d18fc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:45.027800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a22162e-d830-4718-ad92-411018d18fc1 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:45.028321 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a22162e-d830-4718-ad92-411018d18fc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:45.028771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a22162e-d830-4718-ad92-411018d18fc1 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:45.029586 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a22162e-d830-4718-ad92-411018d18fc1 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:45.030130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a22162e-d830-4718-ad92-411018d18fc1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:45.031688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a22162e-d830-4718-ad92-411018d18fc1 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:45.044290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 427/855] 10.222.0.47 () {56 vars in 1126 bytes} [Sun Aug 21 01:05:44 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:45.054784 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfaeb63-7997-4ac4-84cd-d11bd57dfd03 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:45.084087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:45.102018 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc35c600-1f68-4650-8f52-7669dc77fce4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:45.102638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc35c600-1f68-4650-8f52-7669dc77fce4 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:45.102911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc35c600-1f68-4650-8f52-7669dc77fce4 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:45.103182 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc35c600-1f68-4650-8f52-7669dc77fce4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:45.103916 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc35c600-1f68-4650-8f52-7669dc77fce4 None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=06a5972a55824711b06670b925322739) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:45.104258 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc35c600-1f68-4650-8f52-7669dc77fce4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:45.105797 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc35c600-1f68-4650-8f52-7669dc77fce4 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:45.113687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 429/856] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:44 2022] GET /identity/v3/role_assignments?user.id=06a5972a55824711b06670b925322739&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:45.562509 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b901025-a011-46a7-b772-872cd71fceba None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:45.562888 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b901025-a011-46a7-b772-872cd71fceba None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:45.563095 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b901025-a011-46a7-b772-872cd71fceba None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:45.564763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 428/857] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:45.570277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9888221-7b71-4338-9399-2b7c248758a3 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:45.570737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9888221-7b71-4338-9399-2b7c248758a3 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:45.570815 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9888221-7b71-4338-9399-2b7c248758a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:45.589448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-a9888221-7b71-4338-9399-2b7c248758a3 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:45.590627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a9888221-7b71-4338-9399-2b7c248758a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:45.661023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 430/858] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:45 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:45.880728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5ad1b5-7f25-4e02-ad1b-bb756739deeb None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:45.881399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5ad1b5-7f25-4e02-ad1b-bb756739deeb None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:45.881908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5ad1b5-7f25-4e02-ad1b-bb756739deeb None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:45.883777 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 429/859] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:45.889144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84336c2-907d-4b43-b0c4-d2cfb05bf7fc None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:45.889423 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84336c2-907d-4b43-b0c4-d2cfb05bf7fc None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:45.889691 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84336c2-907d-4b43-b0c4-d2cfb05bf7fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:45.908967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-e84336c2-907d-4b43-b0c4-d2cfb05bf7fc None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:45.910137 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e84336c2-907d-4b43-b0c4-d2cfb05bf7fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:45.930308 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5ad1b5-7f25-4e02-ad1b-bb756739deeb None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:45.991085 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 431/860] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:45 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:05:46.045278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5ad1b5-7f25-4e02-ad1b-bb756739deeb None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:46.082667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91af315-6f2f-42be-b2fc-3bd34622774d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:46.083315 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91af315-6f2f-42be-b2fc-3bd34622774d None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:46.083598 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91af315-6f2f-42be-b2fc-3bd34622774d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:46.083870 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91af315-6f2f-42be-b2fc-3bd34622774d None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:46.084538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91af315-6f2f-42be-b2fc-3bd34622774d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:46.085955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91af315-6f2f-42be-b2fc-3bd34622774d None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:46.090513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:46.168051 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 430/861] 10.222.0.47 () {60 vars in 1141 bytes} [Sun Aug 21 01:05:45 2022] POST /identity/v3/services => generated 251 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:05:46.273130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84336c2-907d-4b43-b0c4-d2cfb05bf7fc None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:46.327677 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:46.392220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84336c2-907d-4b43-b0c4-d2cfb05bf7fc None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:46.440127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5555df-c878-4386-b94d-93adbcde0f7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:46.440127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5555df-c878-4386-b94d-93adbcde0f7c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:46.440127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5555df-c878-4386-b94d-93adbcde0f7c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:46.440127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5555df-c878-4386-b94d-93adbcde0f7c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:46.448366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5555df-c878-4386-b94d-93adbcde0f7c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:46.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-ae5555df-c878-4386-b94d-93adbcde0f7c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:46.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:46.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:46.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:46.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:46.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:46.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:46.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:46.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:46.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:46.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:46.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:46.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:46.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:46.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:46.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:46.453952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:46.453952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:46.453952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:46.453952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:46.453952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:46.453952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:46.453952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:46.453952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:46.453952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:46.453952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:46.453952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:46.453952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:46.453952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:46.453952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:46.453952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:46.453952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:46.455824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:46.455824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:46.455824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:46.455824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:46.455824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:46.455824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:46.455824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:46.455824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:46.455824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:46.455824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:46.455824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:05:46.455824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:46.455824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 432/862] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:05:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:46.460823 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91af315-6f2f-42be-b2fc-3bd34622774d None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:46.569004 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91af315-6f2f-42be-b2fc-3bd34622774d None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:46.604592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b767f19-4514-4ae6-b127-38fcce1984c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:46.605194 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b767f19-4514-4ae6-b127-38fcce1984c6 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:46.605467 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b767f19-4514-4ae6-b127-38fcce1984c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:46.605726 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b767f19-4514-4ae6-b127-38fcce1984c6 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:46.606388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b767f19-4514-4ae6-b127-38fcce1984c6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:46.606739 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b767f19-4514-4ae6-b127-38fcce1984c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:46.608134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b767f19-4514-4ae6-b127-38fcce1984c6 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:46.617245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 431/863] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:05:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:46.622687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5555df-c878-4386-b94d-93adbcde0f7c None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:46.731141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5555df-c878-4386-b94d-93adbcde0f7c None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:46.733508 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:46.767678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fa1367-7bc8-4986-aaac-d2c8792f2a32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:46.768399 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fa1367-7bc8-4986-aaac-d2c8792f2a32 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:46.768673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fa1367-7bc8-4986-aaac-d2c8792f2a32 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:46.768936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fa1367-7bc8-4986-aaac-d2c8792f2a32 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:46.779475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fa1367-7bc8-4986-aaac-d2c8792f2a32 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:46.780949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fa1367-7bc8-4986-aaac-d2c8792f2a32 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:46.797292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-29fa1367-7bc8-4986-aaac-d2c8792f2a32 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:46.797292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:46.797292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:46.797292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:46.797292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:46.797292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:46.797292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:46.797292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:46.797292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:46.797292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:46.797292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:46.797292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:46.797292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:46.797292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:46.797292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:46.797292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:46.800169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:46.802475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:46.802475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:46.802475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:46.802475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 433/864] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:46.805313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b767f19-4514-4ae6-b127-38fcce1984c6 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:46.867066 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:46.911000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b767f19-4514-4ae6-b127-38fcce1984c6 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:46.947145 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e528295d-2084-459f-913d-827fdb4440ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:46.947743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e528295d-2084-459f-913d-827fdb4440ec None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:46.948016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e528295d-2084-459f-913d-827fdb4440ec None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:46.948307 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e528295d-2084-459f-913d-827fdb4440ec None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:46.948937 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e528295d-2084-459f-913d-827fdb4440ec None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:46.949231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e528295d-2084-459f-913d-827fdb4440ec None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:46.950661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e528295d-2084-459f-913d-827fdb4440ec None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:46.963329 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 432/865] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:46.968840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fa1367-7bc8-4986-aaac-d2c8792f2a32 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:47.008993 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246c7d1f-5ba7-4587-92e9-d929a2efd634 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:47.009347 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246c7d1f-5ba7-4587-92e9-d929a2efd634 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:47.009530 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246c7d1f-5ba7-4587-92e9-d929a2efd634 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:47.011161 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 433/866] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:47.016701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dc2e28-dbda-48ad-a486-31aa3aed67ce None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:47.017578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dc2e28-dbda-48ad-a486-31aa3aed67ce None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:47.017847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dc2e28-dbda-48ad-a486-31aa3aed67ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:47.037614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-07dc2e28-dbda-48ad-a486-31aa3aed67ce None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:47.038789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-07dc2e28-dbda-48ad-a486-31aa3aed67ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:47.084540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fa1367-7bc8-4986-aaac-d2c8792f2a32 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:47.120622 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 434/867] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:47 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:05:47.121800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7acedc0-b1a2-436f-a6aa-e91bd39fa097 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:47.122076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7acedc0-b1a2-436f-a6aa-e91bd39fa097 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:47.122342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7acedc0-b1a2-436f-a6aa-e91bd39fa097 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:47.123690 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 435/868] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:47 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:47.127086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4097205e-e679-492c-94a4-16900788bf49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:47.127954 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4097205e-e679-492c-94a4-16900788bf49 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:47.128473 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4097205e-e679-492c-94a4-16900788bf49 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:47.128934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4097205e-e679-492c-94a4-16900788bf49 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:47.129443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fff667-7170-4b0f-a255-25edb2f0b871 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:47.129720 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fff667-7170-4b0f-a255-25edb2f0b871 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:47.129989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fff667-7170-4b0f-a255-25edb2f0b871 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:47.138124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4097205e-e679-492c-94a4-16900788bf49 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:47.139831 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4097205e-e679-492c-94a4-16900788bf49 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:47.148508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-d1fff667-7170-4b0f-a255-25edb2f0b871 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:47.149687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-d1fff667-7170-4b0f-a255-25edb2f0b871 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:47.152049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-4097205e-e679-492c-94a4-16900788bf49 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 21 01:05:47.152049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:47.152049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:47.152049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:47.152049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:47.152049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:47.152049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:47.152049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:47.152049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:47.152049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:47.152049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:47.152049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:47.152049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:47.152049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:47.152049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:47.152049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:47.152996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:47.153926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:47.153926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:47.153926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:47.153926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:47.153926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:47.153926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:47.153926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:47.153926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:47.153926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 21 01:05:47.153926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:47.156377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 434/869] 10.222.0.47 () {56 vars in 1117 bytes} [Sun Aug 21 01:05:46 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:47.162186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4097205e-e679-492c-94a4-16900788bf49 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:47.186096 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:47.209566 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:47.248477 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 436/870] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:47 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:05:47.273818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4097205e-e679-492c-94a4-16900788bf49 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:47.309952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde145fa-3051-49f0-9d9f-4365fe4d4760 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:47.310939 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde145fa-3051-49f0-9d9f-4365fe4d4760 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:47.311414 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde145fa-3051-49f0-9d9f-4365fe4d4760 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:47.311879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde145fa-3051-49f0-9d9f-4365fe4d4760 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:47.312813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde145fa-3051-49f0-9d9f-4365fe4d4760 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:47.313384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde145fa-3051-49f0-9d9f-4365fe4d4760 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:47.315086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde145fa-3051-49f0-9d9f-4365fe4d4760 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:47.333605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 435/871] 10.222.0.47 () {56 vars in 1149 bytes} [Sun Aug 21 01:05:47 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:47.341466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fff667-7170-4b0f-a255-25edb2f0b871 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:47.406144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde145fa-3051-49f0-9d9f-4365fe4d4760 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:47.457901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fff667-7170-4b0f-a255-25edb2f0b871 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:47.494411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4d88d9-f995-4802-941f-20655a689512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:47.495075 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4d88d9-f995-4802-941f-20655a689512 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:47.495352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4d88d9-f995-4802-941f-20655a689512 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:47.495631 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4d88d9-f995-4802-941f-20655a689512 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:47.505222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:47.533857 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde145fa-3051-49f0-9d9f-4365fe4d4760 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:47.553458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:47.572342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fbc497-1ee1-461e-9656-7a1e00e698e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:47.572881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fbc497-1ee1-461e-9656-7a1e00e698e7 None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:47.573160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fbc497-1ee1-461e-9656-7a1e00e698e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:47.573432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fbc497-1ee1-461e-9656-7a1e00e698e7 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:47.574078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65fbc497-1ee1-461e-9656-7a1e00e698e7 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:47.575479 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65fbc497-1ee1-461e-9656-7a1e00e698e7 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:47.575584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4d88d9-f995-4802-941f-20655a689512 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:47.577159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4d88d9-f995-4802-941f-20655a689512 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:47.677316 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 436/872] 10.222.0.47 () {60 vars in 1141 bytes} [Sun Aug 21 01:05:47 2022] POST /identity/v3/services => generated 269 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:05:47.678078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fbc497-1ee1-461e-9656-7a1e00e698e7 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:47.684930 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 437/873] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:47 2022] GET /identity/v3/auth/tokens => generated 1262 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:05:47.686499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4d88d9-f995-4802-941f-20655a689512 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:47.788210 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:47.813367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4d88d9-f995-4802-941f-20655a689512 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:47.818277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fbc497-1ee1-461e-9656-7a1e00e698e7 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:47.852760 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:47.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e407852a-9d88-49f3-80bd-f1a8fa3e5236 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:47.855888 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e407852a-9d88-49f3-80bd-f1a8fa3e5236 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:47.856365 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e407852a-9d88-49f3-80bd-f1a8fa3e5236 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:47.856771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e407852a-9d88-49f3-80bd-f1a8fa3e5236 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:47.859338 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ad13c2-0f43-44ea-8a34-b741e69d9609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:47.860046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ad13c2-0f43-44ea-8a34-b741e69d9609 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:47.860374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ad13c2-0f43-44ea-8a34-b741e69d9609 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:47.860724 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ad13c2-0f43-44ea-8a34-b741e69d9609 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:47.861364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ad13c2-0f43-44ea-8a34-b741e69d9609 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:47.862725 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ad13c2-0f43-44ea-8a34-b741e69d9609 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:47.868531 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e407852a-9d88-49f3-80bd-f1a8fa3e5236 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:47.873485 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e407852a-9d88-49f3-80bd-f1a8fa3e5236 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-e4ad13c2-0f43-44ea-8a34-b741e69d9609 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:47.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:47.874516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:47.874516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:47.874516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:47.874516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:47.874516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:47.874516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:47.874516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:05:47.874516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:05:47.874516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:05:47.874516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:05:47.874516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:47.874516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:47.874516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:47.874516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:47.874516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:47.874516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:47.875450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:47.875450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:47.875450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:47.875450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:47.875450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:47.875450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:47.875450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:47.875450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:47.875450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:47.875450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:47.875450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:05:47.875450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:47.875450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 21 01:05:47.875450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:47.876745 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 437/874] 10.222.0.47 () {56 vars in 1126 bytes} [Sun Aug 21 01:05:47 2022] GET /identity/v3/services/object-store => generated 93 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:47.881056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ad13c2-0f43-44ea-8a34-b741e69d9609 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:47.887698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-e407852a-9d88-49f3-80bd-f1a8fa3e5236 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:47.887698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:47.887698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:47.887698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:47.887698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:47.887698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:47.887698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:47.887698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:47.887698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:47.887698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:47.887698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:47.887698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:47.887698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:47.887698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:47.887698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:47.887698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:47.888723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:47.889610 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:47.889610 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:47.889610 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:47.890146 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 438/875] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:47.897766 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e407852a-9d88-49f3-80bd-f1a8fa3e5236 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:48.014279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e407852a-9d88-49f3-80bd-f1a8fa3e5236 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:48.014620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ad13c2-0f43-44ea-8a34-b741e69d9609 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:48.045757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:48.051947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9edb70-7f4e-4639-8ba5-d5d4dc793303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:48.052658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9edb70-7f4e-4639-8ba5-d5d4dc793303 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:48.053001 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9edb70-7f4e-4639-8ba5-d5d4dc793303 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:48.053339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9edb70-7f4e-4639-8ba5-d5d4dc793303 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:48.053517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bede67-989b-4494-b0a9-f4c79cb9eb8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:48.054117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9edb70-7f4e-4639-8ba5-d5d4dc793303 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:48.054439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bede67-989b-4494-b0a9-f4c79cb9eb8c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:48.054550 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9edb70-7f4e-4639-8ba5-d5d4dc793303 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:48.054945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bede67-989b-4494-b0a9-f4c79cb9eb8c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:48.055361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bede67-989b-4494-b0a9-f4c79cb9eb8c None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:48.056082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9edb70-7f4e-4639-8ba5-d5d4dc793303 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:48.061895 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bede67-989b-4494-b0a9-f4c79cb9eb8c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:48.061895 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bede67-989b-4494-b0a9-f4c79cb9eb8c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:48.061895 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bede67-989b-4494-b0a9-f4c79cb9eb8c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:48.068998 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 438/876] 10.222.0.47 () {56 vars in 1122 bytes} [Sun Aug 21 01:05:47 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:48.070030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 439/877] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:48.070279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bede67-989b-4494-b0a9-f4c79cb9eb8c None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:48.076961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9edb70-7f4e-4639-8ba5-d5d4dc793303 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:48.143112 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:48.189973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bede67-989b-4494-b0a9-f4c79cb9eb8c None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:48.196141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9edb70-7f4e-4639-8ba5-d5d4dc793303 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:48.232206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe69aaa-1e77-487c-9246-7a429391807a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:48.232445 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5862b9c7-42e6-461f-8d37-24ba50a3a75f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:48.232999 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5862b9c7-42e6-461f-8d37-24ba50a3a75f None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:48.233274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5862b9c7-42e6-461f-8d37-24ba50a3a75f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:48.233544 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5862b9c7-42e6-461f-8d37-24ba50a3a75f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:48.233939 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe69aaa-1e77-487c-9246-7a429391807a None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:48.234434 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe69aaa-1e77-487c-9246-7a429391807a None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:48.234895 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe69aaa-1e77-487c-9246-7a429391807a None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:48.236538 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe69aaa-1e77-487c-9246-7a429391807a None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:48.237313 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe69aaa-1e77-487c-9246-7a429391807a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:48.238864 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe69aaa-1e77-487c-9246-7a429391807a None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:48.240956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5862b9c7-42e6-461f-8d37-24ba50a3a75f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:48.242371 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5862b9c7-42e6-461f-8d37-24ba50a3a75f None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:48.248288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 439/878] 10.222.0.47 () {56 vars in 1122 bytes} [Sun Aug 21 01:05:48 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:48.249708 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95acfa09-8612-48f7-ab3d-c91c1d36eab9 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:48.249992 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95acfa09-8612-48f7-ab3d-c91c1d36eab9 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:48.250262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95acfa09-8612-48f7-ab3d-c91c1d36eab9 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:48.251923 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 440/879] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:48.257744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ff8d12-fc02-497f-b4a3-f7e12278a321 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:48.258074 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ff8d12-fc02-497f-b4a3-f7e12278a321 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:48.258296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ff8d12-fc02-497f-b4a3-f7e12278a321 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:48.263829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-5862b9c7-42e6-461f-8d37-24ba50a3a75f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:48.263829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:48.263829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:48.263829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:48.263829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:48.263829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:48.263829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:48.263829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:48.263829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:48.263829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:48.263829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:48.263829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:48.263829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:48.263829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:48.263829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:48.265088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:48.266018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:48.266018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:48.266018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:48.266018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:48.266018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:48.266018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:48.266018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:48.266018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:48.270466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 440/880] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:48.276473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5862b9c7-42e6-461f-8d37-24ba50a3a75f None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:48.280297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-51ff8d12-fc02-497f-b4a3-f7e12278a321 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:48.281512 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-51ff8d12-fc02-497f-b4a3-f7e12278a321 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:48.351097 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:48.376366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 441/881] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:48 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:48.423949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5862b9c7-42e6-461f-8d37-24ba50a3a75f None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:48.460055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d17ef9-de09-4af0-9389-3916399fc5a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:48.460813 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d17ef9-de09-4af0-9389-3916399fc5a3 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:48.461089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d17ef9-de09-4af0-9389-3916399fc5a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:48.461400 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d17ef9-de09-4af0-9389-3916399fc5a3 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:48.462091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d17ef9-de09-4af0-9389-3916399fc5a3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:48.462391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d17ef9-de09-4af0-9389-3916399fc5a3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:48.464198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d17ef9-de09-4af0-9389-3916399fc5a3 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:48.478687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 441/882] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:48.484267 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ff8d12-fc02-497f-b4a3-f7e12278a321 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:48.544515 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:48.593791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ff8d12-fc02-497f-b4a3-f7e12278a321 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:48.629154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03a9d66-9314-4757-bcfc-8deda68bcf6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:48.630056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03a9d66-9314-4757-bcfc-8deda68bcf6d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:48.630545 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03a9d66-9314-4757-bcfc-8deda68bcf6d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:48.630992 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03a9d66-9314-4757-bcfc-8deda68bcf6d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:48.631893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03a9d66-9314-4757-bcfc-8deda68bcf6d None admin] RBAC: Adding query filter params (role.id=c2d8bf7283e14857879d03d1f39c0165, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=d7c65195ff8d42958fa54b5f083bf8da) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:48.632399 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03a9d66-9314-4757-bcfc-8deda68bcf6d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:48.633990 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03a9d66-9314-4757-bcfc-8deda68bcf6d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:48.640930 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 442/883] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:48 2022] GET /identity/v3/role_assignments?user.id=d7c65195ff8d42958fa54b5f083bf8da&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=c2d8bf7283e14857879d03d1f39c0165 => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:48.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d17ef9-de09-4af0-9389-3916399fc5a3 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:48.785110 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d17ef9-de09-4af0-9389-3916399fc5a3 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:48.802067 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:48.828208 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc96eb27-c1a7-4ede-a941-56fd105ac2fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:48.828844 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc96eb27-c1a7-4ede-a941-56fd105ac2fb None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:48.829113 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc96eb27-c1a7-4ede-a941-56fd105ac2fb None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:48.829408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc96eb27-c1a7-4ede-a941-56fd105ac2fb None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:48.829985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc96eb27-c1a7-4ede-a941-56fd105ac2fb None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:48.831328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc96eb27-c1a7-4ede-a941-56fd105ac2fb None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-dc96eb27-c1a7-4ede-a941-56fd105ac2fb None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:48.840537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:48.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:48.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:48.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:48.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:48.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:48.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:48.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:05:48.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:05:48.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:05:48.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:05:48.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:48.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:48.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:48.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:48.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:48.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:48.842977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:48.842977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:48.842977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:48.842977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:48.842977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:48.842977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:48.842977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:48.842977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:48.842977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:48.842977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:48.842977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:05:48.842977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:48.842977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 21 01:05:48.842977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:48.842977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 442/884] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:05:48 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:48.851212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03a9d66-9314-4757-bcfc-8deda68bcf6d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:48.962484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03a9d66-9314-4757-bcfc-8deda68bcf6d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:48.979895 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:48.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4641651-3a2a-4cea-94f1-3b126b0d1a99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:48.998388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4641651-3a2a-4cea-94f1-3b126b0d1a99 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:48.998661 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4641651-3a2a-4cea-94f1-3b126b0d1a99 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:49.000218 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4641651-3a2a-4cea-94f1-3b126b0d1a99 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:49.000218 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4641651-3a2a-4cea-94f1-3b126b0d1a99 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:49.000218 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4641651-3a2a-4cea-94f1-3b126b0d1a99 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:49.001870 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4641651-3a2a-4cea-94f1-3b126b0d1a99 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:49.006912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 443/885] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:05:48 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:49.012600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc96eb27-c1a7-4ede-a941-56fd105ac2fb None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:49.118534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc96eb27-c1a7-4ede-a941-56fd105ac2fb None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:49.153821 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a11dbe-2992-4b57-8bf8-ccc601c711e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:49.154435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a11dbe-2992-4b57-8bf8-ccc601c711e5 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:49.154710 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a11dbe-2992-4b57-8bf8-ccc601c711e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:49.154980 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a11dbe-2992-4b57-8bf8-ccc601c711e5 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:49.155611 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a11dbe-2992-4b57-8bf8-ccc601c711e5 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:49.155909 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a11dbe-2992-4b57-8bf8-ccc601c711e5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:49.157335 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a11dbe-2992-4b57-8bf8-ccc601c711e5 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:49.167313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 443/886] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:05:49 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:49.169174 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4641651-3a2a-4cea-94f1-3b126b0d1a99 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:49.274514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4641651-3a2a-4cea-94f1-3b126b0d1a99 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:49.310095 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b010be-1508-4507-9e1e-c8cceb386e5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:49.310762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b010be-1508-4507-9e1e-c8cceb386e5a None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:49.311042 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b010be-1508-4507-9e1e-c8cceb386e5a None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:49.311303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b010be-1508-4507-9e1e-c8cceb386e5a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:49.312041 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b010be-1508-4507-9e1e-c8cceb386e5a None admin] RBAC: Adding query filter params (interface=public, service_id=d854a89ad22e4dba85309826cbcc0f2b, region_id=RegionOne) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:49.312371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b010be-1508-4507-9e1e-c8cceb386e5a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:49.313863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b010be-1508-4507-9e1e-c8cceb386e5a None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:49.319915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 444/887] 10.222.0.47 () {56 vars in 1251 bytes} [Sun Aug 21 01:05:49 2022] GET /identity/v3/endpoints?service_id=d854a89ad22e4dba85309826cbcc0f2b&interface=public®ion_id=RegionOne => generated 194 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:49.325416 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a11dbe-2992-4b57-8bf8-ccc601c711e5 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:49.369363 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:49.435534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a11dbe-2992-4b57-8bf8-ccc601c711e5 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:49.471047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a54443-362e-48bb-b280-5dd8bbf712af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:49.471660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a54443-362e-48bb-b280-5dd8bbf712af None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:49.471937 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a54443-362e-48bb-b280-5dd8bbf712af None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:49.472234 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a54443-362e-48bb-b280-5dd8bbf712af None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:49.472834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a54443-362e-48bb-b280-5dd8bbf712af None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:49.474196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a54443-362e-48bb-b280-5dd8bbf712af None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:49.484469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 444/888] 10.222.0.47 () {56 vars in 1087 bytes} [Sun Aug 21 01:05:49 2022] GET /identity/v3/services => generated 1268 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:05:49.751597 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c30087-2b84-452f-9906-b20960f8b616 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:49.751998 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c30087-2b84-452f-9906-b20960f8b616 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:49.752209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c30087-2b84-452f-9906-b20960f8b616 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:49.753768 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 445/889] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:49.758954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115930ee-69a6-44e5-afab-895fbb4e832d None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:49.759234 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115930ee-69a6-44e5-afab-895fbb4e832d None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:49.759507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115930ee-69a6-44e5-afab-895fbb4e832d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:49.779754 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-115930ee-69a6-44e5-afab-895fbb4e832d None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:49.780961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-115930ee-69a6-44e5-afab-895fbb4e832d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:49.855996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 445/890] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:49 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:50.132230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c30087-2b84-452f-9906-b20960f8b616 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:50.181154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:50.238307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c30087-2b84-452f-9906-b20960f8b616 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:50.260428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68135b5c-2823-4c94-8bda-7f5ff8e9f8f0 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:50.260760 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68135b5c-2823-4c94-8bda-7f5ff8e9f8f0 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:50.260977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68135b5c-2823-4c94-8bda-7f5ff8e9f8f0 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:50.262579 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 446/891] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:50.267648 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cd67af-1e35-4d41-81f5-57560ccdb3dd None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:50.267927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cd67af-1e35-4d41-81f5-57560ccdb3dd None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:50.268229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cd67af-1e35-4d41-81f5-57560ccdb3dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:50.273239 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3286a467-6ff7-48c4-b37a-94a2b8006ffb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:50.274102 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3286a467-6ff7-48c4-b37a-94a2b8006ffb None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:50.274692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3286a467-6ff7-48c4-b37a-94a2b8006ffb None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:50.275146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3286a467-6ff7-48c4-b37a-94a2b8006ffb None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:50.276030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3286a467-6ff7-48c4-b37a-94a2b8006ffb None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:50.277396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3286a467-6ff7-48c4-b37a-94a2b8006ffb None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:50.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-3286a467-6ff7-48c4-b37a-94a2b8006ffb None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 21 01:05:50.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:50.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:05:50.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:05:50.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:05:50.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:05:50.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:05:50.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:50.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 21 01:05:50.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:50.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:50.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:50.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:50.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:50.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:50.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:50.288027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:50.289081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:50.289081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:50.289081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:50.289081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:50.289081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:50.289081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:50.289081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:50.289081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:50.289081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:50.289081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:50.289081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:05:50.289081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:50.289081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 21 01:05:50.289081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:50.289081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 446/892] 10.222.0.47 () {56 vars in 1132 bytes} [Sun Aug 21 01:05:50 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:50.291238 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-27cd67af-1e35-4d41-81f5-57560ccdb3dd None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:50.292715 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-27cd67af-1e35-4d41-81f5-57560ccdb3dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:50.293975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3286a467-6ff7-48c4-b37a-94a2b8006ffb None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:50.330985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:50.392223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 447/893] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:50 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:05:50.426716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3286a467-6ff7-48c4-b37a-94a2b8006ffb None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:50.437276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:50.464504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c25da3d-38c7-4e66-a660-3e08d30ad510 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:50.465172 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c25da3d-38c7-4e66-a660-3e08d30ad510 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:50.465451 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c25da3d-38c7-4e66-a660-3e08d30ad510 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:50.465728 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c25da3d-38c7-4e66-a660-3e08d30ad510 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:50.466358 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c25da3d-38c7-4e66-a660-3e08d30ad510 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:50.466657 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c25da3d-38c7-4e66-a660-3e08d30ad510 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:50.468014 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c25da3d-38c7-4e66-a660-3e08d30ad510 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:50.473462 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 447/894] 10.222.0.47 () {56 vars in 1126 bytes} [Sun Aug 21 01:05:50 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:50.479206 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cd67af-1e35-4d41-81f5-57560ccdb3dd None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:50.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cd67af-1e35-4d41-81f5-57560ccdb3dd None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:50.628075 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef589755-abb8-46e6-91df-b2be19fb46c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:50.628703 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef589755-abb8-46e6-91df-b2be19fb46c2 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:50.628980 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef589755-abb8-46e6-91df-b2be19fb46c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:50.629247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef589755-abb8-46e6-91df-b2be19fb46c2 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:50.629878 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef589755-abb8-46e6-91df-b2be19fb46c2 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:50.630179 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef589755-abb8-46e6-91df-b2be19fb46c2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:50.631540 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef589755-abb8-46e6-91df-b2be19fb46c2 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:50.637135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 448/895] 10.222.0.47 () {56 vars in 1126 bytes} [Sun Aug 21 01:05:50 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:50.643703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c25da3d-38c7-4e66-a660-3e08d30ad510 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:50.714482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef589755-abb8-46e6-91df-b2be19fb46c2 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:50.756670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c25da3d-38c7-4e66-a660-3e08d30ad510 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:50.760456 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:50.800718 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c2d975-452f-4fa6-adbd-e5195f2623c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:50.801401 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c2d975-452f-4fa6-adbd-e5195f2623c1 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:50.801741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c2d975-452f-4fa6-adbd-e5195f2623c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:50.802042 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c2d975-452f-4fa6-adbd-e5195f2623c1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:50.802850 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c2d975-452f-4fa6-adbd-e5195f2623c1 None admin] RBAC: Adding query filter params (interface=public, service_id=4203a3ddc9be4bd28dbe9b0d532f754c, region_id=RegionOne) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:50.803233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c2d975-452f-4fa6-adbd-e5195f2623c1 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:50.804717 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c2d975-452f-4fa6-adbd-e5195f2623c1 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:50.810426 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 448/896] 10.222.0.47 () {56 vars in 1251 bytes} [Sun Aug 21 01:05:50 2022] GET /identity/v3/endpoints?service_id=4203a3ddc9be4bd28dbe9b0d532f754c&interface=public®ion_id=RegionOne => generated 194 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:50.816672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c2d975-452f-4fa6-adbd-e5195f2623c1 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:50.835468 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef589755-abb8-46e6-91df-b2be19fb46c2 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:50.853118 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:50.878458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fbb2a1-7e0e-4c3f-98e0-a63f7dddd4fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:50.879154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fbb2a1-7e0e-4c3f-98e0-a63f7dddd4fb None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:50.879490 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fbb2a1-7e0e-4c3f-98e0-a63f7dddd4fb None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:50.879834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fbb2a1-7e0e-4c3f-98e0-a63f7dddd4fb None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:50.880575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fbb2a1-7e0e-4c3f-98e0-a63f7dddd4fb None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:50.881990 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fbb2a1-7e0e-4c3f-98e0-a63f7dddd4fb None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:50.923081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c2d975-452f-4fa6-adbd-e5195f2623c1 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:50.948134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 449/897] 10.222.0.47 () {60 vars in 1141 bytes} [Sun Aug 21 01:05:50 2022] POST /identity/v3/services => generated 241 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:05:50.948134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdcec0a-4a71-4624-9ac8-3bc2b9bc6117 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:50.948134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdcec0a-4a71-4624-9ac8-3bc2b9bc6117 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:50.948134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdcec0a-4a71-4624-9ac8-3bc2b9bc6117 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:50.948134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 450/898] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:50.955151 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5a91b6-f91e-4b9f-bc6f-4f5a02594aec None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:50.955503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5a91b6-f91e-4b9f-bc6f-4f5a02594aec None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:50.955845 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5a91b6-f91e-4b9f-bc6f-4f5a02594aec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:50.978959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-ee5a91b6-f91e-4b9f-bc6f-4f5a02594aec None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:50.980466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ee5a91b6-f91e-4b9f-bc6f-4f5a02594aec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:50.983555 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d925125-b513-48ac-aa57-11aee002c816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:50.984431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d925125-b513-48ac-aa57-11aee002c816 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:50.984851 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d925125-b513-48ac-aa57-11aee002c816 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:50.985328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d925125-b513-48ac-aa57-11aee002c816 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:50.986089 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d925125-b513-48ac-aa57-11aee002c816 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:50.987606 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d925125-b513-48ac-aa57-11aee002c816 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:50.994803 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 449/899] 10.222.0.47 () {56 vars in 1087 bytes} [Sun Aug 21 01:05:50 2022] GET /identity/v3/services => generated 1497 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:05:51.076288 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 451/900] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:50 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:51.362879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d925125-b513-48ac-aa57-11aee002c816 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:51.468565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:51.475500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0efade-3413-4797-8d06-4b88831be19a None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:51.476006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0efade-3413-4797-8d06-4b88831be19a None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:51.476006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0efade-3413-4797-8d06-4b88831be19a None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:51.477532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 452/901] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:51.477660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d925125-b513-48ac-aa57-11aee002c816 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:51.483301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf4a95a-dfdb-45bc-b115-224f3f4ee436 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:51.483573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf4a95a-dfdb-45bc-b115-224f3f4ee436 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:51.483849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf4a95a-dfdb-45bc-b115-224f3f4ee436 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:51.519695 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4fa541-0c35-4922-b305-b92717f8200b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:51.519695 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4fa541-0c35-4922-b305-b92717f8200b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:51.519695 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4fa541-0c35-4922-b305-b92717f8200b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:51.519695 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4fa541-0c35-4922-b305-b92717f8200b None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:51.520199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-faf4a95a-dfdb-45bc-b115-224f3f4ee436 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:51.523990 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-faf4a95a-dfdb-45bc-b115-224f3f4ee436 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:51.531621 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4fa541-0c35-4922-b305-b92717f8200b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:51.534624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:51.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-ba4fa541-0c35-4922-b305-b92717f8200b None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 21 01:05:51.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:51.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:51.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:51.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:51.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:51.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:51.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:51.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:51.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:51.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:51.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:51.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:51.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:51.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:51.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:51.540955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:51.540955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:51.540955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:51.540955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:51.540955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:51.540955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:51.540955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:51.540955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:51.540955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:51.540955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:51.540955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:51.540955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:51.540955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:51.540955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:51.540955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:51.540955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:51.541962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:51.541962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:51.541962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:51.541962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:51.541962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:51.541962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:51.541962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:51.541962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:51.541962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:51.541962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:51.541962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 21 01:05:51.541962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:51.544723 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 450/902] 10.222.0.47 () {56 vars in 1120 bytes} [Sun Aug 21 01:05:51 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:51.550690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4fa541-0c35-4922-b305-b92717f8200b None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:51.599010 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 453/903] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:51 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:51.659470 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4fa541-0c35-4922-b305-b92717f8200b None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:51.695530 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594437f3-0878-4904-ac2d-c52f199532ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:51.696154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594437f3-0878-4904-ac2d-c52f199532ee None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:51.696437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594437f3-0878-4904-ac2d-c52f199532ee None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:51.696705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594437f3-0878-4904-ac2d-c52f199532ee None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:51.697357 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594437f3-0878-4904-ac2d-c52f199532ee None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:51.697650 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594437f3-0878-4904-ac2d-c52f199532ee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:51.698991 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594437f3-0878-4904-ac2d-c52f199532ee None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:51.708366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 451/904] 10.222.0.47 () {56 vars in 1115 bytes} [Sun Aug 21 01:05:51 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:51.713957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf4a95a-dfdb-45bc-b115-224f3f4ee436 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:51.817732 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf4a95a-dfdb-45bc-b115-224f3f4ee436 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:51.854577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9feada9-8f2f-4475-9f32-3115f13805af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:51.855128 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9feada9-8f2f-4475-9f32-3115f13805af None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:51.855399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9feada9-8f2f-4475-9f32-3115f13805af None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:51.855707 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9feada9-8f2f-4475-9f32-3115f13805af None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:51.867387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9feada9-8f2f-4475-9f32-3115f13805af None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:51.868840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9feada9-8f2f-4475-9f32-3115f13805af None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:51.875668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:51.883815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-f9feada9-8f2f-4475-9f32-3115f13805af None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:51.883815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:51.883815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:51.883815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:51.883815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:51.883815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:51.883815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:51.883815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:51.883815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:51.883815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:51.883815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:51.883815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:51.883815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:51.883815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:51.883815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:51.883815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:51.884900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:51.885822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:51.885822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:51.885822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:51.887979 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 454/905] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:51.891604 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594437f3-0878-4904-ac2d-c52f199532ee None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:51.901237 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9feada9-8f2f-4475-9f32-3115f13805af None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:51.913907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:52.009757 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594437f3-0878-4904-ac2d-c52f199532ee None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:52.030645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9feada9-8f2f-4475-9f32-3115f13805af None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:52.055065 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983f86bd-a589-4657-9273-7d0dc86a89df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:52.055877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983f86bd-a589-4657-9273-7d0dc86a89df None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:52.056182 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983f86bd-a589-4657-9273-7d0dc86a89df None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:52.056458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983f86bd-a589-4657-9273-7d0dc86a89df None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:52.057031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983f86bd-a589-4657-9273-7d0dc86a89df None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:52.059010 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983f86bd-a589-4657-9273-7d0dc86a89df None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:52.069228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7a6ae9-24a5-4ca5-9052-ad42b13cddf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-983f86bd-a589-4657-9273-7d0dc86a89df None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:52.069571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:52.070567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:52.070567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:52.070567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:52.070567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:52.070567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:52.070567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:52.070567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:05:52.070567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:05:52.070567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:05:52.070567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:05:52.070567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:52.070567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:52.070567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:52.070567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:52.070567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:52.070567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:52.076614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:52.076614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:52.076614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:52.076614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:52.076614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:52.076614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:52.076614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:52.076614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:52.076614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:52.076614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:52.076614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:05:52.076614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:52.076614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 21 01:05:52.076614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:52.076614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 452/906] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:05:51 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:52.077458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7a6ae9-24a5-4ca5-9052-ad42b13cddf6 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:52.077458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7a6ae9-24a5-4ca5-9052-ad42b13cddf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:52.077458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7a6ae9-24a5-4ca5-9052-ad42b13cddf6 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:52.077458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7a6ae9-24a5-4ca5-9052-ad42b13cddf6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:52.077458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7a6ae9-24a5-4ca5-9052-ad42b13cddf6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:52.077458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7a6ae9-24a5-4ca5-9052-ad42b13cddf6 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:52.077820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983f86bd-a589-4657-9273-7d0dc86a89df None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:52.091981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 455/907] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:52.098085 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7a6ae9-24a5-4ca5-9052-ad42b13cddf6 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:52.162419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:52.183150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:52.203257 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983f86bd-a589-4657-9273-7d0dc86a89df None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:52.210270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7a6ae9-24a5-4ca5-9052-ad42b13cddf6 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:52.240043 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459fe284-4855-4ee0-97ee-aaf0328c8e5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:52.240863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459fe284-4855-4ee0-97ee-aaf0328c8e5b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:52.241447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459fe284-4855-4ee0-97ee-aaf0328c8e5b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:52.241729 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459fe284-4855-4ee0-97ee-aaf0328c8e5b None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:52.242427 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459fe284-4855-4ee0-97ee-aaf0328c8e5b None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:52.242662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459fe284-4855-4ee0-97ee-aaf0328c8e5b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:52.244034 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459fe284-4855-4ee0-97ee-aaf0328c8e5b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:52.254956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 453/908] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:05:52 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:52.255174 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189bbdbe-779a-4e89-888e-e01c6f08cee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:52.255670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189bbdbe-779a-4e89-888e-e01c6f08cee2 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:52.255942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189bbdbe-779a-4e89-888e-e01c6f08cee2 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:52.257127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459fe284-4855-4ee0-97ee-aaf0328c8e5b None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:52.257649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189bbdbe-779a-4e89-888e-e01c6f08cee2 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:52.267739 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189bbdbe-779a-4e89-888e-e01c6f08cee2 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:52.269869 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189bbdbe-779a-4e89-888e-e01c6f08cee2 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:52.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-189bbdbe-779a-4e89-888e-e01c6f08cee2 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 21 01:05:52.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:52.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:52.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:52.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:52.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:52.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:52.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:52.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:52.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:52.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:52.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:52.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:52.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:52.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:52.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:52.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:52.288428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:52.288428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:52.288428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:52.288428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:52.288428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:52.288428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:52.288428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:52.288428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:52.288428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 21 01:05:52.288428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:52.292185 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 456/909] 10.222.0.47 () {56 vars in 1117 bytes} [Sun Aug 21 01:05:52 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:52.294554 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189bbdbe-779a-4e89-888e-e01c6f08cee2 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:52.387637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459fe284-4855-4ee0-97ee-aaf0328c8e5b None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:52.408257 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189bbdbe-779a-4e89-888e-e01c6f08cee2 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:52.427647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:52.429116 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77220d3a-fe80-4851-895f-60aff9c55b22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:52.429989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77220d3a-fe80-4851-895f-60aff9c55b22 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:52.430828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77220d3a-fe80-4851-895f-60aff9c55b22 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:52.431321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77220d3a-fe80-4851-895f-60aff9c55b22 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:52.432202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77220d3a-fe80-4851-895f-60aff9c55b22 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:52.432723 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77220d3a-fe80-4851-895f-60aff9c55b22 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:52.434381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77220d3a-fe80-4851-895f-60aff9c55b22 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:52.440777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 454/910] 10.222.0.47 () {56 vars in 1114 bytes} [Sun Aug 21 01:05:52 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:52.447846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77220d3a-fe80-4851-895f-60aff9c55b22 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:52.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d192a1b-3024-4ae0-b322-39182c74f718 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:52.454155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d192a1b-3024-4ae0-b322-39182c74f718 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:52.454642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d192a1b-3024-4ae0-b322-39182c74f718 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:52.455114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d192a1b-3024-4ae0-b322-39182c74f718 None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:52.456044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d192a1b-3024-4ae0-b322-39182c74f718 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:52.456566 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d192a1b-3024-4ae0-b322-39182c74f718 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:52.458199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d192a1b-3024-4ae0-b322-39182c74f718 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:52.468027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:52.488442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 457/911] 10.222.0.47 () {56 vars in 1149 bytes} [Sun Aug 21 01:05:52 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:52.491265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d192a1b-3024-4ae0-b322-39182c74f718 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:52.581839 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77220d3a-fe80-4851-895f-60aff9c55b22 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:52.604655 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d192a1b-3024-4ae0-b322-39182c74f718 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:52.624911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87f8124-b65f-4d61-84c8-3c6243a8cf18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:52.626069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87f8124-b65f-4d61-84c8-3c6243a8cf18 None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:52.626736 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87f8124-b65f-4d61-84c8-3c6243a8cf18 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:52.627227 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87f8124-b65f-4d61-84c8-3c6243a8cf18 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:52.628092 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87f8124-b65f-4d61-84c8-3c6243a8cf18 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:52.629808 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87f8124-b65f-4d61-84c8-3c6243a8cf18 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:52.645596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66322170-864e-4ace-92ad-b2fbec02c9a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:52.646228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66322170-864e-4ace-92ad-b2fbec02c9a3 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:52.646531 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66322170-864e-4ace-92ad-b2fbec02c9a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:52.646801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66322170-864e-4ace-92ad-b2fbec02c9a3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:52.695957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:52.720199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66322170-864e-4ace-92ad-b2fbec02c9a3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:52.721825 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66322170-864e-4ace-92ad-b2fbec02c9a3 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:52.732540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 455/912] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:05:52 2022] POST /identity/v3/endpoints => generated 346 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:05:52.840591 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 458/913] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:05:52 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:05:52.848043 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87f8124-b65f-4d61-84c8-3c6243a8cf18 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:52.931655 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:52.980474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87f8124-b65f-4d61-84c8-3c6243a8cf18 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:53.023609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130fb9f0-d433-4137-be41-e623d787e499 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:53.024287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130fb9f0-d433-4137-be41-e623d787e499 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:53.024564 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130fb9f0-d433-4137-be41-e623d787e499 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:53.024885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130fb9f0-d433-4137-be41-e623d787e499 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:53.036929 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130fb9f0-d433-4137-be41-e623d787e499 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:53.038384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130fb9f0-d433-4137-be41-e623d787e499 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:53.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-130fb9f0-d433-4137-be41-e623d787e499 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:53.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:53.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:53.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:53.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:53.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:53.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:53.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:53.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:53.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:53.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:53.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:53.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:53.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:53.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:53.054756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:53.056730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:53.056730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:53.056730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:53.056730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 456/914] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:53.061838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66322170-864e-4ace-92ad-b2fbec02c9a3 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:53.126783 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f1c21-9094-450b-a432-08e9c2ea9305 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:53.127437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f1c21-9094-450b-a432-08e9c2ea9305 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:53.128005 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f1c21-9094-450b-a432-08e9c2ea9305 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:53.130005 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 457/915] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:53 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:05:53.135620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202f0dc8-df24-4e94-a612-4c5fb2d05492 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:53.135620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202f0dc8-df24-4e94-a612-4c5fb2d05492 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:53.135620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202f0dc8-df24-4e94-a612-4c5fb2d05492 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:53.144914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:53.156004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-202f0dc8-df24-4e94-a612-4c5fb2d05492 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:53.160147 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-202f0dc8-df24-4e94-a612-4c5fb2d05492 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:53.173469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66322170-864e-4ace-92ad-b2fbec02c9a3 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:53.226081 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c836990d-d70e-4e61-b5fb-be935a914024 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:53.226705 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c836990d-d70e-4e61-b5fb-be935a914024 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:53.226962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c836990d-d70e-4e61-b5fb-be935a914024 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:53.227228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c836990d-d70e-4e61-b5fb-be935a914024 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:53.228348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c836990d-d70e-4e61-b5fb-be935a914024 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:53.228702 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c836990d-d70e-4e61-b5fb-be935a914024 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:53.230318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c836990d-d70e-4e61-b5fb-be935a914024 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:53.245495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 459/917] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:53.245658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 458/917] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:53 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:05:53.250398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c836990d-d70e-4e61-b5fb-be935a914024 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:53.255820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8504a8d6-6a0f-4e83-98ed-5b01fbf3809d None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:53.256229 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8504a8d6-6a0f-4e83-98ed-5b01fbf3809d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:53.256665 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8504a8d6-6a0f-4e83-98ed-5b01fbf3809d None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:53.258696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 459/918] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:05:53.263211 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a00aa61-97e0-4e48-8b10-0b6fd46620a2 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:53.263766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a00aa61-97e0-4e48-8b10-0b6fd46620a2 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:53.264140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a00aa61-97e0-4e48-8b10-0b6fd46620a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:53.283936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-8a00aa61-97e0-4e48-8b10-0b6fd46620a2 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:53.285457 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8a00aa61-97e0-4e48-8b10-0b6fd46620a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:53.330775 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:53.362926 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c836990d-d70e-4e61-b5fb-be935a914024 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:53.368508 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 460/919] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:53 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:05:53.398757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079dbb82-02d9-4377-ae14-b38002d0b682 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:53.399604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079dbb82-02d9-4377-ae14-b38002d0b682 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:53.399977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079dbb82-02d9-4377-ae14-b38002d0b682 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:53.400383 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079dbb82-02d9-4377-ae14-b38002d0b682 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:53.406639 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:53.407893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079dbb82-02d9-4377-ae14-b38002d0b682 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:53.409673 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079dbb82-02d9-4377-ae14-b38002d0b682 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:53.421298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-079dbb82-02d9-4377-ae14-b38002d0b682 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:53.421298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:53.421298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:53.421298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:53.421298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:53.421298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:53.421298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:53.421298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:53.421298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:53.421298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:53.421298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:53.421298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:53.421298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:53.421298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:53.421298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:53.422419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:53.423584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:53.423584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:53.423584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:53.423584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:53.423584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:53.423584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:53.423584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:53.423584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:53.425269 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 460/920] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:53.430950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a00aa61-97e0-4e48-8b10-0b6fd46620a2 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:53.521718 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079dbb82-02d9-4377-ae14-b38002d0b682 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:53.539736 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a00aa61-97e0-4e48-8b10-0b6fd46620a2 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:53.578628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054869c1-0142-4bb6-99bd-adb0156e69da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:53.579426 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054869c1-0142-4bb6-99bd-adb0156e69da None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:53.579818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054869c1-0142-4bb6-99bd-adb0156e69da None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:53.580232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054869c1-0142-4bb6-99bd-adb0156e69da None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:53.581252 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054869c1-0142-4bb6-99bd-adb0156e69da None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:53.581388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054869c1-0142-4bb6-99bd-adb0156e69da None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:53.583099 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054869c1-0142-4bb6-99bd-adb0156e69da None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:53.595576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 461/921] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:53.601148 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054869c1-0142-4bb6-99bd-adb0156e69da None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:53.637879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:53.658811 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079dbb82-02d9-4377-ae14-b38002d0b682 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:53.694323 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7109d8-5c17-4bec-8bb4-f0785edd28b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:53.694972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7109d8-5c17-4bec-8bb4-f0785edd28b1 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:53.695245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7109d8-5c17-4bec-8bb4-f0785edd28b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:53.695580 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7109d8-5c17-4bec-8bb4-f0785edd28b1 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:53.696076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7109d8-5c17-4bec-8bb4-f0785edd28b1 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:53.697443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7109d8-5c17-4bec-8bb4-f0785edd28b1 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:53.707279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054869c1-0142-4bb6-99bd-adb0156e69da None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-7d7109d8-5c17-4bec-8bb4-f0785edd28b1 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 21 01:05:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:05:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:05:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:05:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:05:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:05:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 21 01:05:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:53.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:53.709413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:53.709413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:53.709413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:53.709413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:53.709413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:53.709413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:53.709413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:53.709413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:53.709413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:53.709413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:53.709413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:05:53.709413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:53.709413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 21 01:05:53.709413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:53.709413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 461/922] 10.222.0.47 () {56 vars in 1132 bytes} [Sun Aug 21 01:05:53 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:53.710198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7109d8-5c17-4bec-8bb4-f0785edd28b1 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:53.746655 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2cf01e-9827-40e0-be12-d9622651d609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:53.747263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2cf01e-9827-40e0-be12-d9622651d609 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:53.747550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2cf01e-9827-40e0-be12-d9622651d609 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:53.747815 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2cf01e-9827-40e0-be12-d9622651d609 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:53.748762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2cf01e-9827-40e0-be12-d9622651d609 None admin] RBAC: Adding query filter params (role.id=830cf269498649038ec8a27a3712fc60, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=d7c65195ff8d42958fa54b5f083bf8da) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:53.748888 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2cf01e-9827-40e0-be12-d9622651d609 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:53.750274 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2cf01e-9827-40e0-be12-d9622651d609 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:53.756126 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 462/923] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:05:53 2022] GET /identity/v3/role_assignments?user.id=d7c65195ff8d42958fa54b5f083bf8da&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=830cf269498649038ec8a27a3712fc60 => generated 259 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:53.756850 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2cf01e-9827-40e0-be12-d9622651d609 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:53.791936 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:53.846412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7109d8-5c17-4bec-8bb4-f0785edd28b1 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:53.892466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84fba52-8f91-48b8-992a-b70370514ed1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:53.893015 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84fba52-8f91-48b8-992a-b70370514ed1 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:53.893294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84fba52-8f91-48b8-992a-b70370514ed1 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:53.893562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84fba52-8f91-48b8-992a-b70370514ed1 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:53.894152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84fba52-8f91-48b8-992a-b70370514ed1 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:53.895535 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84fba52-8f91-48b8-992a-b70370514ed1 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:53.905445 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2cf01e-9827-40e0-be12-d9622651d609 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-d84fba52-8f91-48b8-992a-b70370514ed1 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:53.906658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:53.907806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:53.907806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:53.907806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:53.907806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:53.907806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:53.907806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:53.907806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:05:53.907806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:05:53.907806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:05:53.907806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:05:53.907806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:53.907806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:53.907806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:53.907806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:53.907806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:53.907806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:53.909057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:53.909057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:53.909057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:53.909057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:53.909057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:53.909057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:53.909057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:53.909057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:53.909057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:53.909057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:53.909057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:05:53.909057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:53.909057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 21 01:05:53.909057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:53.910775 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:53.911616 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 462/924] 10.222.0.47 () {56 vars in 1126 bytes} [Sun Aug 21 01:05:53 2022] GET /identity/v3/services/object-store => generated 93 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:53.914524 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84fba52-8f91-48b8-992a-b70370514ed1 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:53.961184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff9f719-5846-42b6-ace8-13f33674e858 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:53.961959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff9f719-5846-42b6-ace8-13f33674e858 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:53.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff9f719-5846-42b6-ace8-13f33674e858 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:53.962734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff9f719-5846-42b6-ace8-13f33674e858 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:53.963469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff9f719-5846-42b6-ace8-13f33674e858 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:53.963907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff9f719-5846-42b6-ace8-13f33674e858 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:53.965433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff9f719-5846-42b6-ace8-13f33674e858 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:53.972915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 463/925] 10.222.0.47 () {56 vars in 1126 bytes} [Sun Aug 21 01:05:53 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:53.981136 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff9f719-5846-42b6-ace8-13f33674e858 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:54.053079 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84fba52-8f91-48b8-992a-b70370514ed1 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:54.081029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:54.097308 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975cf424-9db8-4034-91f0-1438e9754342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:54.098504 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975cf424-9db8-4034-91f0-1438e9754342 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:54.098957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975cf424-9db8-4034-91f0-1438e9754342 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:54.099396 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975cf424-9db8-4034-91f0-1438e9754342 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:54.100311 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975cf424-9db8-4034-91f0-1438e9754342 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:54.101371 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975cf424-9db8-4034-91f0-1438e9754342 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:54.103206 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975cf424-9db8-4034-91f0-1438e9754342 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:54.111737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 463/926] 10.222.0.47 () {56 vars in 1122 bytes} [Sun Aug 21 01:05:53 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:54.114452 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff9f719-5846-42b6-ace8-13f33674e858 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:54.117932 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975cf424-9db8-4034-91f0-1438e9754342 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:54.152271 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5386164-1ac0-490f-9bfc-db9793496c74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:54.152955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5386164-1ac0-490f-9bfc-db9793496c74 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:54.153334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5386164-1ac0-490f-9bfc-db9793496c74 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:54.153667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5386164-1ac0-490f-9bfc-db9793496c74 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:54.154477 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5386164-1ac0-490f-9bfc-db9793496c74 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:54.154934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5386164-1ac0-490f-9bfc-db9793496c74 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:54.156623 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5386164-1ac0-490f-9bfc-db9793496c74 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:54.162534 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 464/927] 10.222.0.47 () {56 vars in 1126 bytes} [Sun Aug 21 01:05:53 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:54.168783 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5386164-1ac0-490f-9bfc-db9793496c74 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:54.205957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:54.238809 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975cf424-9db8-4034-91f0-1438e9754342 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:54.283756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0d13ee-93dd-4cc3-b3c4-0c67c3f89679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:54.284982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0d13ee-93dd-4cc3-b3c4-0c67c3f89679 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:54.285438 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0d13ee-93dd-4cc3-b3c4-0c67c3f89679 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:54.286128 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0d13ee-93dd-4cc3-b3c4-0c67c3f89679 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:54.286908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0d13ee-93dd-4cc3-b3c4-0c67c3f89679 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:54.287514 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0d13ee-93dd-4cc3-b3c4-0c67c3f89679 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:54.289663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0d13ee-93dd-4cc3-b3c4-0c67c3f89679 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:54.291581 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5386164-1ac0-490f-9bfc-db9793496c74 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:54.302269 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 464/928] 10.222.0.47 () {56 vars in 1122 bytes} [Sun Aug 21 01:05:54 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:54.306587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0d13ee-93dd-4cc3-b3c4-0c67c3f89679 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:54.333609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273c9a41-7767-4d32-b430-ab9a99c60aa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:54.334497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273c9a41-7767-4d32-b430-ab9a99c60aa8 None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:54.334940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273c9a41-7767-4d32-b430-ab9a99c60aa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:54.335382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273c9a41-7767-4d32-b430-ab9a99c60aa8 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:54.336220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273c9a41-7767-4d32-b430-ab9a99c60aa8 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:54.340985 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273c9a41-7767-4d32-b430-ab9a99c60aa8 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:54.364905 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:54.413591 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0d13ee-93dd-4cc3-b3c4-0c67c3f89679 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:54.452605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 465/929] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:05:54 2022] POST /identity/v3/endpoints => generated 347 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:05:54.455054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751cf17d-a1f0-417b-b34e-147683d715cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:54.455816 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751cf17d-a1f0-417b-b34e-147683d715cf None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:54.456262 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751cf17d-a1f0-417b-b34e-147683d715cf None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:54.456615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751cf17d-a1f0-417b-b34e-147683d715cf None admin] PATH_INFO: `/v3/endpoints` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:54.461441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751cf17d-a1f0-417b-b34e-147683d715cf None admin] RBAC: Adding query filter params (interface=public, service_id=44a81662972148fc9d3d68ee5d8c93c9, region_id=RegionOne) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:54.461441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751cf17d-a1f0-417b-b34e-147683d715cf None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:54.461441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751cf17d-a1f0-417b-b34e-147683d715cf None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:54.466023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 465/930] 10.222.0.47 () {56 vars in 1251 bytes} [Sun Aug 21 01:05:54 2022] GET /identity/v3/endpoints?service_id=44a81662972148fc9d3d68ee5d8c93c9&interface=public®ion_id=RegionOne => generated 194 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:54.472450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273c9a41-7767-4d32-b430-ab9a99c60aa8 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:54.570091 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:54.591655 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273c9a41-7767-4d32-b430-ab9a99c60aa8 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:54.635099 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c625c7f-a276-4146-9d1b-4d2c2801be01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:54.635910 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c625c7f-a276-4146-9d1b-4d2c2801be01 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:54.636326 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c625c7f-a276-4146-9d1b-4d2c2801be01 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:54.636790 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c625c7f-a276-4146-9d1b-4d2c2801be01 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:54.637805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c625c7f-a276-4146-9d1b-4d2c2801be01 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:54.639304 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c625c7f-a276-4146-9d1b-4d2c2801be01 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:54.646773 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 466/931] 10.222.0.47 () {56 vars in 1087 bytes} [Sun Aug 21 01:05:54 2022] GET /identity/v3/services => generated 1497 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:05:54.873131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a6bd07-4cbe-494c-821b-66ac82c07d83 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:54.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a6bd07-4cbe-494c-821b-66ac82c07d83 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:54.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a6bd07-4cbe-494c-821b-66ac82c07d83 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:54.875178 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 466/932] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:54.880551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f757be-bc19-4b04-ba64-6331314254b5 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:54.880834 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f757be-bc19-4b04-ba64-6331314254b5 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:54.881108 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f757be-bc19-4b04-ba64-6331314254b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:54.899013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-60f757be-bc19-4b04-ba64-6331314254b5 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:54.900227 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-60f757be-bc19-4b04-ba64-6331314254b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:54.968040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 467/933] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:54 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:55.237744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a6bd07-4cbe-494c-821b-66ac82c07d83 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:55.354710 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a6bd07-4cbe-494c-821b-66ac82c07d83 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:55.368730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:55.391929 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d1bd9c-4146-44e9-8e3c-538aae9ff738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:55.392904 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d1bd9c-4146-44e9-8e3c-538aae9ff738 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:55.393425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d1bd9c-4146-44e9-8e3c-538aae9ff738 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:55.393908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d1bd9c-4146-44e9-8e3c-538aae9ff738 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:55.394751 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d1bd9c-4146-44e9-8e3c-538aae9ff738 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:55.396610 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d1bd9c-4146-44e9-8e3c-538aae9ff738 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:55.399536 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: WARNING keystone.server.flask.common [None req-85d1bd9c-4146-44e9-8e3c-538aae9ff738 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Aug 21 01:05:55.449036 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.sql.core [None req-85d1bd9c-4146-44e9-8e3c-538aae9ff738 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 21 01:05:55.449036 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 21 01:05:55.449036 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [parameters: {'id': '6efcbe9a31094ba09a1307b6850e612c', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 21 01:05:55.449036 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81804) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 21 01:05:55.467045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-85d1bd9c-4146-44e9-8e3c-538aae9ff738 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 6efcbe9a31094ba09a1307b6850e612c.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 6efcbe9a31094ba09a1307b6850e612c. Aug 21 01:05:55.467045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:55.467045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 21 01:05:55.467045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 21 01:05:55.467045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 21 01:05:55.467045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 21 01:05:55.467045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 21 01:05:55.467045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application result = self._query(query) Aug 21 01:05:55.467045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 21 01:05:55.467045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application conn.query(q) Aug 21 01:05:55.467045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 21 01:05:55.467045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 21 01:05:55.467045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application result.read() Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Aug 21 01:05:55.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return project_ref.to_dict() Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application next(self.gen) Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application yield resource Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application next(self.gen) Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self.session.rollback() Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self.force_reraise() Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise self.value Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application session.commit() Aug 21 01:05:55.468955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._prepare_impl() Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self.session.flush() Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._flush(objects) Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application compat.raise_( Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application flush_context.execute() Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 21 01:05:55.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rec.execute(self) Aug 21 01:05:55.470851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 21 01:05:55.470851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 21 01:05:55.470851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 21 01:05:55.470851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 21 01:05:55.470851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 21 01:05:55.470851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 21 01:05:55.470851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 21 01:05:55.470851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 21 01:05:55.470851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 21 01:05:55.470851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 21 01:05:55.470851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 21 01:05:55.470851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 21 01:05:55.470851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 21 01:05:55.470851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 21 01:05:55.470851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application result = self._query(query) Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application conn.query(q) Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application result.read() Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 21 01:05:55.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 21 01:05:55.474414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 21 01:05:55.474414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 21 01:05:55.474414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 21 01:05:55.474414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 21 01:05:55.474414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 21 01:05:55.474414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 21 01:05:55.474414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 21 01:05:55.474414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 21 01:05:55.474414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [parameters: {'id': '6efcbe9a31094ba09a1307b6850e612c', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 21 01:05:55.474414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 21 01:05:55.474414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:55.474414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:55.474414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:55.474414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:55.474414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:55.476355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:55.478606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:55.478606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Aug 21 01:05:55.478606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Aug 21 01:05:55.478606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:55.478606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:55.478606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Aug 21 01:05:55.478606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.Conflict( Aug 21 01:05:55.478606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 6efcbe9a31094ba09a1307b6850e612c. Aug 21 01:05:55.478606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:55.481324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 467/934] 10.222.0.47 () {60 vars in 1140 bytes} [Sun Aug 21 01:05:55 2022] POST /identity/v3/projects => generated 262 bytes in 244 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:55.487041 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f757be-bc19-4b04-ba64-6331314254b5 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:55.568160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:55.591820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f757be-bc19-4b04-ba64-6331314254b5 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:55.627698 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86478c2-4947-4743-9f42-ee24f0318ac1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:55.628359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86478c2-4947-4743-9f42-ee24f0318ac1 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:55.628639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86478c2-4947-4743-9f42-ee24f0318ac1 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:55.628911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86478c2-4947-4743-9f42-ee24f0318ac1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:55.636005 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86478c2-4947-4743-9f42-ee24f0318ac1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:55.637541 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86478c2-4947-4743-9f42-ee24f0318ac1 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:55.649542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-f86478c2-4947-4743-9f42-ee24f0318ac1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:55.649542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:55.649542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:55.649542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:55.649542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:55.649542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:55.649542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:55.649542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:55.649542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:55.649542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:55.649542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:55.649542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:55.649542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:55.649542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:55.649542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:55.650486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:55.651805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:55.651805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:55.651805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:55.651805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:55.651805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:55.651805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:55.651805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:55.651805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:55.651805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 468/935] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:55.658059 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d1bd9c-4146-44e9-8e3c-538aae9ff738 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:55.746045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc98c23-32c1-4205-882e-dbe086f829a5 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:55.746396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc98c23-32c1-4205-882e-dbe086f829a5 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:55.746598 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc98c23-32c1-4205-882e-dbe086f829a5 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:55.748537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 469/936] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:55.754750 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3140fd8-8adb-4c3f-b7ef-7c705b86d5e1 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:55.755138 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3140fd8-8adb-4c3f-b7ef-7c705b86d5e1 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:55.755508 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3140fd8-8adb-4c3f-b7ef-7c705b86d5e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:55.772584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d1bd9c-4146-44e9-8e3c-538aae9ff738 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:55.779041 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-c3140fd8-8adb-4c3f-b7ef-7c705b86d5e1 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:55.780855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-c3140fd8-8adb-4c3f-b7ef-7c705b86d5e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:55.813574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfdae84-ed23-4d44-ad2d-0be775a4653c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:55.814437 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfdae84-ed23-4d44-ad2d-0be775a4653c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:55.814890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfdae84-ed23-4d44-ad2d-0be775a4653c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:55.815589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfdae84-ed23-4d44-ad2d-0be775a4653c None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:55.816442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfdae84-ed23-4d44-ad2d-0be775a4653c None admin] RBAC: Adding query filter params (name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:55.816897 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfdae84-ed23-4d44-ad2d-0be775a4653c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:55.818740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfdae84-ed23-4d44-ad2d-0be775a4653c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:55.841621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 468/937] 10.222.0.47 () {56 vars in 1112 bytes} [Sun Aug 21 01:05:55 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:55.860756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 470/938] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:55 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:56.132662 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfdae84-ed23-4d44-ad2d-0be775a4653c None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:56.163266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:56.243344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfdae84-ed23-4d44-ad2d-0be775a4653c None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:56.278806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed835b2-80a2-4261-a74c-9167694a771c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:56.279438 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed835b2-80a2-4261-a74c-9167694a771c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:56.279762 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed835b2-80a2-4261-a74c-9167694a771c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:56.280043 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed835b2-80a2-4261-a74c-9167694a771c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:56.291084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed835b2-80a2-4261-a74c-9167694a771c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:05:56.295911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-4ed835b2-80a2-4261-a74c-9167694a771c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 21 01:05:56.295911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:56.295911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:56.295911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:56.295911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:56.295911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:56.295911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:56.295911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:56.295911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:56.295911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:56.295911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:56.295911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:56.295911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:05:56.295911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:05:56.295911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:05:56.295911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:05:56.296917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:05:56.296917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:05:56.296917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:05:56.296917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:05:56.296917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:56.296917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:56.296917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:56.296917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:56.296917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:56.296917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:56.296917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:56.296917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:56.296917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:56.296917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:56.296917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:56.296917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:56.298404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:56.298404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:56.298404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:56.298404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:56.298404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:05:56.298404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:05:56.298404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:05:56.298404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:05:56.298404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:05:56.298404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:05:56.298404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 21 01:05:56.298404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:56.298404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 469/939] 10.222.0.47 () {56 vars in 1120 bytes} [Sun Aug 21 01:05:56 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:56.305680 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3140fd8-8adb-4c3f-b7ef-7c705b86d5e1 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:56.385840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:56.420718 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3140fd8-8adb-4c3f-b7ef-7c705b86d5e1 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:56.457550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1eb198e-265e-4b0b-a9b7-18b39579ed26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:56.458160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eb198e-265e-4b0b-a9b7-18b39579ed26 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:56.458434 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eb198e-265e-4b0b-a9b7-18b39579ed26 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:56.458701 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eb198e-265e-4b0b-a9b7-18b39579ed26 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:56.459363 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1eb198e-265e-4b0b-a9b7-18b39579ed26 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:56.459658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1eb198e-265e-4b0b-a9b7-18b39579ed26 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:56.461049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1eb198e-265e-4b0b-a9b7-18b39579ed26 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:56.469812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 471/940] 10.222.0.47 () {56 vars in 1115 bytes} [Sun Aug 21 01:05:56 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:56.475248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed835b2-80a2-4261-a74c-9167694a771c None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:56.572837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db15b862-3cfe-4759-b3b5-a1fe7fe5b902 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:56.573354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db15b862-3cfe-4759-b3b5-a1fe7fe5b902 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:56.573354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db15b862-3cfe-4759-b3b5-a1fe7fe5b902 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:56.575004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 472/941] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:56.575473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:56.580262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5095985e-2c47-46f0-ab80-4ba30d84d2a6 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:56.580545 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5095985e-2c47-46f0-ab80-4ba30d84d2a6 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:56.580818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5095985e-2c47-46f0-ab80-4ba30d84d2a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:56.587809 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed835b2-80a2-4261-a74c-9167694a771c None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:56.599036 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-5095985e-2c47-46f0-ab80-4ba30d84d2a6 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:56.600395 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5095985e-2c47-46f0-ab80-4ba30d84d2a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:56.623152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6d8a14-b3e5-490c-acb7-682186fdcb0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:56.623779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6d8a14-b3e5-490c-acb7-682186fdcb0e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:56.624052 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6d8a14-b3e5-490c-acb7-682186fdcb0e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:56.624410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6d8a14-b3e5-490c-acb7-682186fdcb0e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:56.638908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6d8a14-b3e5-490c-acb7-682186fdcb0e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:56.640315 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6d8a14-b3e5-490c-acb7-682186fdcb0e None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:56.656191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-8b6d8a14-b3e5-490c-acb7-682186fdcb0e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:56.656191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:56.656191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:56.656191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:56.656191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:56.656191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:56.656191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:56.656191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:56.656191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:56.656191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:56.656191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:56.656191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:56.656191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:56.656191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:56.656191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:56.656191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:56.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:56.658054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:56.658054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:56.658054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:56.658640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 470/942] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:56.659781 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8b1dd9-a46b-426d-bcd3-703057b2e493 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:56.660419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8b1dd9-a46b-426d-bcd3-703057b2e493 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:56.660697 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8b1dd9-a46b-426d-bcd3-703057b2e493 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:56.664141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 471/943] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:56.664141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8b1dd9-a46b-426d-bcd3-703057b2e493 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:56.674689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 473/944] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:56 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:05:56.675931 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d318f6-de8e-461f-a355-611c3f1cb621 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:56.676590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d318f6-de8e-461f-a355-611c3f1cb621 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:56.676590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d318f6-de8e-461f-a355-611c3f1cb621 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:56.704516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-57d318f6-de8e-461f-a355-611c3f1cb621 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:56.705973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-57d318f6-de8e-461f-a355-611c3f1cb621 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:56.772329 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:56.775663 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 474/945] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:56 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:56.796319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8b1dd9-a46b-426d-bcd3-703057b2e493 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:56.832617 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611de189-acd2-4a8f-b807-47943323416b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:56.833282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611de189-acd2-4a8f-b807-47943323416b None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:56.833555 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611de189-acd2-4a8f-b807-47943323416b None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:56.833822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611de189-acd2-4a8f-b807-47943323416b None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:56.834454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611de189-acd2-4a8f-b807-47943323416b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:56.834749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611de189-acd2-4a8f-b807-47943323416b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:56.836176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611de189-acd2-4a8f-b807-47943323416b None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:56.846253 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:56.849927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 472/946] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:56.860945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d318f6-de8e-461f-a355-611c3f1cb621 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:56.965265 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d318f6-de8e-461f-a355-611c3f1cb621 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:57.001284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea6378a-1bfa-433e-a500-135d4dc534c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:57.002527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea6378a-1bfa-433e-a500-135d4dc534c8 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:57.002921 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611de189-acd2-4a8f-b807-47943323416b None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:57.003064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea6378a-1bfa-433e-a500-135d4dc534c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:57.003440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea6378a-1bfa-433e-a500-135d4dc534c8 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:57.013836 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea6378a-1bfa-433e-a500-135d4dc534c8 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:57.015704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea6378a-1bfa-433e-a500-135d4dc534c8 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:57.030235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-5ea6378a-1bfa-433e-a500-135d4dc534c8 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 21 01:05:57.030235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:57.030235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:57.030235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:57.030235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:57.030235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:57.030235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:57.030235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:57.030235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:57.030235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:57.030235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:57.030235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:57.030235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:57.030235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:57.030235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:57.030235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:57.031490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:57.032684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:57.032684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:57.032684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:57.032684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:57.032684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:57.032684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:57.032684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:57.032684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:57.032684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 21 01:05:57.032684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:57.034840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 475/947] 10.222.0.47 () {56 vars in 1117 bytes} [Sun Aug 21 01:05:56 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:57.040974 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea6378a-1bfa-433e-a500-135d4dc534c8 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:57.097405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:57.117737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611de189-acd2-4a8f-b807-47943323416b None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:57.152270 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea6378a-1bfa-433e-a500-135d4dc534c8 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:57.153235 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b16772-403a-4fb2-b079-d1cb778fec6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:57.153972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b16772-403a-4fb2-b079-d1cb778fec6a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:57.154242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b16772-403a-4fb2-b079-d1cb778fec6a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:57.154502 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b16772-403a-4fb2-b079-d1cb778fec6a None admin] PATH_INFO: `/v3/services/compute` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:57.155423 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b16772-403a-4fb2-b079-d1cb778fec6a None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:57.157005 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b16772-403a-4fb2-b079-d1cb778fec6a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-44b16772-403a-4fb2-b079-d1cb778fec6a None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:57.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:57.166669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:57.166669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:57.166669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:57.166669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:57.166669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:57.166669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:57.166669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:05:57.166669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:05:57.166669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:05:57.166669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:05:57.166669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:57.166669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:57.166669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:57.166669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:57.166669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:57.166669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:57.167518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:57.167518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:57.167518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:57.167518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:57.167518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:57.167518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:57.167518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:57.167518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:57.167518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:57.167518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:57.167518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:05:57.167518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:57.167518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 21 01:05:57.167518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:57.167518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 473/948] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:57 2022] GET /identity/v3/services/compute => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:05:57.168354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b16772-403a-4fb2-b079-d1cb778fec6a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:57.204120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1381dae-3ea1-4b55-a5ee-32edace1fccb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:57.204882 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1381dae-3ea1-4b55-a5ee-32edace1fccb None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:57.205158 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1381dae-3ea1-4b55-a5ee-32edace1fccb None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:57.205428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1381dae-3ea1-4b55-a5ee-32edace1fccb None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:57.206149 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1381dae-3ea1-4b55-a5ee-32edace1fccb None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:57.206449 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1381dae-3ea1-4b55-a5ee-32edace1fccb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:57.207856 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1381dae-3ea1-4b55-a5ee-32edace1fccb None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:57.234225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 476/949] 10.222.0.47 () {56 vars in 1149 bytes} [Sun Aug 21 01:05:57 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:05:57.234225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1381dae-3ea1-4b55-a5ee-32edace1fccb None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:57.245784 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:57.288748 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b16772-403a-4fb2-b079-d1cb778fec6a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:57.325981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fcb404-49ce-443b-b4f4-63e68d20fb06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:57.326897 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fcb404-49ce-443b-b4f4-63e68d20fb06 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:57.327171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fcb404-49ce-443b-b4f4-63e68d20fb06 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:57.327442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fcb404-49ce-443b-b4f4-63e68d20fb06 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:57.328021 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fcb404-49ce-443b-b4f4-63e68d20fb06 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:57.329441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fcb404-49ce-443b-b4f4-63e68d20fb06 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-86fcb404-49ce-443b-b4f4-63e68d20fb06 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:57.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:57.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:57.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:57.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:57.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:57.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:57.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:57.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:05:57.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:05:57.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:05:57.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:05:57.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:57.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:57.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:57.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:57.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:57.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:57.340422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:57.340422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:57.340422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:57.340422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:57.340422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:57.340422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:57.340422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:57.340422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:57.340422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:57.340422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:57.340422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:05:57.340422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:05:57.340422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 21 01:05:57.340422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:57.344683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 474/950] 10.222.0.47 () {56 vars in 1126 bytes} [Sun Aug 21 01:05:57 2022] GET /identity/v3/services/object-store => generated 93 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:57.345581 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fcb404-49ce-443b-b4f4-63e68d20fb06 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:57.353147 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1381dae-3ea1-4b55-a5ee-32edace1fccb None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:57.363461 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:57.390514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd78195-e698-4faa-a684-3d2d78624396 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:57.391151 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd78195-e698-4faa-a684-3d2d78624396 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:57.391426 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd78195-e698-4faa-a684-3d2d78624396 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:57.391696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd78195-e698-4faa-a684-3d2d78624396 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:57.392345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd78195-e698-4faa-a684-3d2d78624396 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:57.392759 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd78195-e698-4faa-a684-3d2d78624396 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:57.395105 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd78195-e698-4faa-a684-3d2d78624396 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:57.405811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 477/951] 10.222.0.47 () {56 vars in 1112 bytes} [Sun Aug 21 01:05:57 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:57.405811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd78195-e698-4faa-a684-3d2d78624396 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:57.453189 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fcb404-49ce-443b-b4f4-63e68d20fb06 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:57.479778 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:57.489533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2606dbd4-5e0c-4b12-9463-f2f585aaf4b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:57.489533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2606dbd4-5e0c-4b12-9463-f2f585aaf4b2 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:57.489766 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2606dbd4-5e0c-4b12-9463-f2f585aaf4b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:57.489975 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2606dbd4-5e0c-4b12-9463-f2f585aaf4b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:57.501798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2606dbd4-5e0c-4b12-9463-f2f585aaf4b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:57.503242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2606dbd4-5e0c-4b12-9463-f2f585aaf4b2 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:57.514343 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd78195-e698-4faa-a684-3d2d78624396 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:57.518276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-2606dbd4-5e0c-4b12-9463-f2f585aaf4b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:57.518276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:57.518276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:57.518276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:57.518276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:57.518276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:57.518276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:57.518276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:57.518276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:57.518276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:57.518276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:57.518276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:57.518276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:05:57.518276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:05:57.518276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:05:57.518276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:57.519296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:05:57.525413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:05:57.525413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:05:57.525413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:57.525413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 475/952] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:05:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:57.525413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2606dbd4-5e0c-4b12-9463-f2f585aaf4b2 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:57.555327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08582d7-37aa-4b1e-b469-1023c1a6d508 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:57.555944 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08582d7-37aa-4b1e-b469-1023c1a6d508 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:57.556264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08582d7-37aa-4b1e-b469-1023c1a6d508 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:57.556534 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08582d7-37aa-4b1e-b469-1023c1a6d508 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:57.557156 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08582d7-37aa-4b1e-b469-1023c1a6d508 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:57.557457 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08582d7-37aa-4b1e-b469-1023c1a6d508 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:57.558803 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08582d7-37aa-4b1e-b469-1023c1a6d508 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:57.564785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 478/953] 10.222.0.47 () {56 vars in 1122 bytes} [Sun Aug 21 01:05:57 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:57.565673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08582d7-37aa-4b1e-b469-1023c1a6d508 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:57.600524 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:57.634894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2606dbd4-5e0c-4b12-9463-f2f585aaf4b2 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:57.673111 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a5cc3c-2637-430c-82fc-3232e953fd93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:57.673354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08582d7-37aa-4b1e-b469-1023c1a6d508 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:57.673718 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a5cc3c-2637-430c-82fc-3232e953fd93 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:57.673986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a5cc3c-2637-430c-82fc-3232e953fd93 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:57.674249 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a5cc3c-2637-430c-82fc-3232e953fd93 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:57.674977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a5cc3c-2637-430c-82fc-3232e953fd93 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:57.675303 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a5cc3c-2637-430c-82fc-3232e953fd93 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:57.676753 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a5cc3c-2637-430c-82fc-3232e953fd93 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:57.688154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 476/954] 10.222.0.47 () {56 vars in 1112 bytes} [Sun Aug 21 01:05:57 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:57.688154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a5cc3c-2637-430c-82fc-3232e953fd93 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:57.717781 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d8a650-4b8b-48bb-b936-5c705f446eb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:57.718698 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d8a650-4b8b-48bb-b936-5c705f446eb4 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:57.719693 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d8a650-4b8b-48bb-b936-5c705f446eb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:57.720141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d8a650-4b8b-48bb-b936-5c705f446eb4 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:57.720933 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d8a650-4b8b-48bb-b936-5c705f446eb4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:57.721376 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d8a650-4b8b-48bb-b936-5c705f446eb4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:57.722928 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d8a650-4b8b-48bb-b936-5c705f446eb4 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:57.736619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 479/955] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:05:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:57.742418 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d8a650-4b8b-48bb-b936-5c705f446eb4 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:57.754177 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:57.826102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a5cc3c-2637-430c-82fc-3232e953fd93 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:57.868407 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77e356f-bfe2-4a97-8cd1-e630553034a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:57.869035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77e356f-bfe2-4a97-8cd1-e630553034a0 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:57.869311 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77e356f-bfe2-4a97-8cd1-e630553034a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:57.869582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77e356f-bfe2-4a97-8cd1-e630553034a0 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:57.870218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77e356f-bfe2-4a97-8cd1-e630553034a0 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:57.870704 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77e356f-bfe2-4a97-8cd1-e630553034a0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:57.870908 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:57.871873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77e356f-bfe2-4a97-8cd1-e630553034a0 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:57.877586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d8a650-4b8b-48bb-b936-5c705f446eb4 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:57.878832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 477/956] 10.222.0.47 () {56 vars in 1122 bytes} [Sun Aug 21 01:05:57 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:57.879906 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87554d68-65b9-4f07-8369-10c3649d21c2 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:57.880194 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87554d68-65b9-4f07-8369-10c3649d21c2 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:57.884994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87554d68-65b9-4f07-8369-10c3649d21c2 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:57.884994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 478/957] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:57.886053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87554d68-65b9-4f07-8369-10c3649d21c2 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:57.917233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df35b37-d127-481e-b0e7-9cc9ff61dc2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:57.917868 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df35b37-d127-481e-b0e7-9cc9ff61dc2e None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:57.918147 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df35b37-d127-481e-b0e7-9cc9ff61dc2e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:57.918419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df35b37-d127-481e-b0e7-9cc9ff61dc2e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:57.925900 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df35b37-d127-481e-b0e7-9cc9ff61dc2e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:57.927332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df35b37-d127-481e-b0e7-9cc9ff61dc2e None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:57.938697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-7df35b37-d127-481e-b0e7-9cc9ff61dc2e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:57.938697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:57.938697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:57.938697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:57.938697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:57.938697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:57.938697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:57.938697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:57.938697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:57.938697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:57.938697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:57.938697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:57.938697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:05:57.938697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:05:57.938697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:57.946054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:05:57.946054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:05:57.946054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:05:57.946054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:05:57.946054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:05:57.946054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:05:57.946054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:05:57.946054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:57.946054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 480/958] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:05:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:57.946054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738adfd6-5100-499d-8fa0-c6133d1a74d2 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:57.946054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738adfd6-5100-499d-8fa0-c6133d1a74d2 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:57.946836 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738adfd6-5100-499d-8fa0-c6133d1a74d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:57.961262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-738adfd6-5100-499d-8fa0-c6133d1a74d2 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:57.962458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-738adfd6-5100-499d-8fa0-c6133d1a74d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:58.006833 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87554d68-65b9-4f07-8369-10c3649d21c2 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:58.017806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:58.040281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 481/959] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:57 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:05:58.040281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738adfd6-5100-499d-8fa0-c6133d1a74d2 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:58.050514 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23514487-6ac5-4e9f-8dcb-ff9d5c8af399 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:58.051200 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23514487-6ac5-4e9f-8dcb-ff9d5c8af399 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:58.051543 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23514487-6ac5-4e9f-8dcb-ff9d5c8af399 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:58.051890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23514487-6ac5-4e9f-8dcb-ff9d5c8af399 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:58.052735 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23514487-6ac5-4e9f-8dcb-ff9d5c8af399 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:58.054156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23514487-6ac5-4e9f-8dcb-ff9d5c8af399 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:58.135202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:58.147657 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 479/960] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:05:57 2022] POST /identity/v3/endpoints => generated 349 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:05:58.167865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738adfd6-5100-499d-8fa0-c6133d1a74d2 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:58.216469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6e9147-2155-43be-9b9e-c520639038a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:58.217099 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6e9147-2155-43be-9b9e-c520639038a1 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:58.217367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6e9147-2155-43be-9b9e-c520639038a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:58.217639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6e9147-2155-43be-9b9e-c520639038a1 None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:58.218335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6e9147-2155-43be-9b9e-c520639038a1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:58.218641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6e9147-2155-43be-9b9e-c520639038a1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:58.220869 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6e9147-2155-43be-9b9e-c520639038a1 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:58.234784 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 482/961] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:05:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:58.240600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23514487-6ac5-4e9f-8dcb-ff9d5c8af399 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:58.330899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6e9147-2155-43be-9b9e-c520639038a1 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:58.356746 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23514487-6ac5-4e9f-8dcb-ff9d5c8af399 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:58.394279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3062e8-0e82-4588-a697-842d5ef697fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:58.395084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3062e8-0e82-4588-a697-842d5ef697fc None admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:58.395479 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3062e8-0e82-4588-a697-842d5ef697fc None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:58.395848 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3062e8-0e82-4588-a697-842d5ef697fc None admin] PATH_INFO: `/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/d7c65195ff8d42958fa54b5f083bf8da/roles/830cf269498649038ec8a27a3712fc60` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:58.398324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3062e8-0e82-4588-a697-842d5ef697fc None admin] RBAC: Authorizing `identity:create_grant(project_id=75a3bf3f3a4244d18fde47d492657db4, user_id=d7c65195ff8d42958fa54b5f083bf8da, role_id=830cf269498649038ec8a27a3712fc60)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:58.400054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3062e8-0e82-4588-a697-842d5ef697fc None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:58.445696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6e9147-2155-43be-9b9e-c520639038a1 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:58.481017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fb83b0-1740-42ce-b43b-e175c7945d72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:58.481903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fb83b0-1740-42ce-b43b-e175c7945d72 None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:58.482348 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fb83b0-1740-42ce-b43b-e175c7945d72 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:58.482806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fb83b0-1740-42ce-b43b-e175c7945d72 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:58.483678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fb83b0-1740-42ce-b43b-e175c7945d72 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:58.485293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fb83b0-1740-42ce-b43b-e175c7945d72 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:58.488005 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: WARNING keystone.server.flask.common [None req-f5fb83b0-1740-42ce-b43b-e175c7945d72 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Aug 21 01:05:58.499468 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 480/962] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:05:58 2022] PUT /identity/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/d7c65195ff8d42958fa54b5f083bf8da/roles/830cf269498649038ec8a27a3712fc60 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:05:58.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.sql.core [None req-f5fb83b0-1740-42ce-b43b-e175c7945d72 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 21 01:05:58.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 21 01:05:58.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [parameters: {'user_id': '1644eed0238f44cf89dfb305a1d213f8', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 21 01:05:58.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81805) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 21 01:05:58.562528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-f5fb83b0-1740-42ce-b43b-e175c7945d72 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 21 01:05:58.562528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:58.562528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 21 01:05:58.562528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 21 01:05:58.562528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 21 01:05:58.562528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 21 01:05:58.562528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 21 01:05:58.562528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application result = self._query(query) Aug 21 01:05:58.562528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 21 01:05:58.562528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application conn.query(q) Aug 21 01:05:58.562528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 21 01:05:58.562528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 21 01:05:58.562528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 21 01:05:58.562528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application result.read() Aug 21 01:05:58.562528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 21 01:05:58.564234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application next(self.gen) Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application yield resource Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application next(self.gen) Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self.session.rollback() Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self.force_reraise() Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise self.value Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application session.commit() Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 21 01:05:58.565726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._prepare_impl() Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self.session.flush() Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._flush(objects) Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application compat.raise_( Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application flush_context.execute() Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rec.execute(self) Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 21 01:05:58.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 21 01:05:58.568140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 21 01:05:58.568140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 21 01:05:58.568140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Aug 21 01:05:58.568140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application result = connection._execute_20( Aug 21 01:05:58.568140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 21 01:05:58.568140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 21 01:05:58.568140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 21 01:05:58.568140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 21 01:05:58.568140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 21 01:05:58.568140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 21 01:05:58.568140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 21 01:05:58.568140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 21 01:05:58.568140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 21 01:05:58.568140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 21 01:05:58.568140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 21 01:05:58.568140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception Aug 21 01:05:58.569262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 21 01:05:58.569262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 21 01:05:58.569262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 21 01:05:58.569262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 21 01:05:58.569262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 21 01:05:58.569262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application result = self._query(query) Aug 21 01:05:58.569262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 21 01:05:58.569262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application conn.query(q) Aug 21 01:05:58.569262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 21 01:05:58.569262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 21 01:05:58.569262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 21 01:05:58.569262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application result.read() Aug 21 01:05:58.569262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 21 01:05:58.569262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 21 01:05:58.569262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 21 01:05:58.569262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 21 01:05:58.570285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 21 01:05:58.570285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 21 01:05:58.570285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 21 01:05:58.570285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 21 01:05:58.570285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 21 01:05:58.570285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 21 01:05:58.570285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [parameters: {'user_id': '1644eed0238f44cf89dfb305a1d213f8', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 21 01:05:58.570285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 21 01:05:58.570285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:58.570285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:05:58.570285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:58.570285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:58.570285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:58.570285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:58.570285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Aug 21 01:05:58.571246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Aug 21 01:05:58.572177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Aug 21 01:05:58.572177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Aug 21 01:05:58.572177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 21 01:05:58.572177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 21 01:05:58.572177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 21 01:05:58.572177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:05:58.574346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 483/963] 10.222.0.47 () {60 vars in 1131 bytes} [Sun Aug 21 01:05:58 2022] POST /identity/v3/users => generated 160 bytes in 244 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:58.581411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3062e8-0e82-4588-a697-842d5ef697fc None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:58.611989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:58.696259 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3062e8-0e82-4588-a697-842d5ef697fc None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:58.736309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8fa724-80dd-4b4e-9aea-018e113f03e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:58.736947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8fa724-80dd-4b4e-9aea-018e113f03e8 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:58.737223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8fa724-80dd-4b4e-9aea-018e113f03e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:58.737495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8fa724-80dd-4b4e-9aea-018e113f03e8 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:58.745618 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8fa724-80dd-4b4e-9aea-018e113f03e8 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:58.747061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8fa724-80dd-4b4e-9aea-018e113f03e8 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:58.759084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-4d8fa724-80dd-4b4e-9aea-018e113f03e8 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 21 01:05:58.759084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:05:58.759084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:05:58.759084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:05:58.759084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:05:58.759084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:05:58.759084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:05:58.759084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:05:58.759084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:05:58.759084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:05:58.759084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:05:58.759084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:05:58.759084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:05:58.759084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:05:58.759084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:05:58.759084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:05:58.760316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:05:58.761512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:05:58.761512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:05:58.761512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:05:58.761512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:05:58.761512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:05:58.761512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:05:58.761512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:05:58.761512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:05:58.761512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 21 01:05:58.761512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:05:58.761512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 481/964] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:05:58 2022] GET /identity/v3/users/cinder => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:05:58.768125 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fb83b0-1740-42ce-b43b-e175c7945d72 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:05:58.771149 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:58.877703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fb83b0-1740-42ce-b43b-e175c7945d72 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:05:58.913154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5b87e6-4d60-4c5c-be15-3dc96f614400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:05:58.913814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5b87e6-4d60-4c5c-be15-3dc96f614400 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:58.914090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5b87e6-4d60-4c5c-be15-3dc96f614400 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:58.914401 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5b87e6-4d60-4c5c-be15-3dc96f614400 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:58.915091 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5b87e6-4d60-4c5c-be15-3dc96f614400 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:05:58.915387 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5b87e6-4d60-4c5c-be15-3dc96f614400 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:05:58.917095 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5b87e6-4d60-4c5c-be15-3dc96f614400 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:05:58.955812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 484/965] 10.222.0.47 () {56 vars in 1101 bytes} [Sun Aug 21 01:05:58 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:05:59.696358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8a7739-e630-425b-8f7d-4afb8bd22ba3 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:59.696854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8a7739-e630-425b-8f7d-4afb8bd22ba3 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:59.696854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8a7739-e630-425b-8f7d-4afb8bd22ba3 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:59.698160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 482/966] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:05:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:05:59.704245 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681f6d31-2087-4f78-8596-2647786ef753 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:05:59.704535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681f6d31-2087-4f78-8596-2647786ef753 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:05:59.704805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681f6d31-2087-4f78-8596-2647786ef753 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:05:59.725385 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-681f6d31-2087-4f78-8596-2647786ef753 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:05:59.726900 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-681f6d31-2087-4f78-8596-2647786ef753 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:05:59.785019 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:05:59.796088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 485/967] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:05:59 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:00.068235 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8a7739-e630-425b-8f7d-4afb8bd22ba3 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:00.134448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f83390-6938-4c94-914a-406dacec8eed None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:00.134731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f83390-6938-4c94-914a-406dacec8eed None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:00.135030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f83390-6938-4c94-914a-406dacec8eed None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:00.136833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 486/968] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:06:00.141788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161b1700-7005-4cae-8220-f40abc01efd6 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:00.142071 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161b1700-7005-4cae-8220-f40abc01efd6 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:00.142344 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161b1700-7005-4cae-8220-f40abc01efd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:00.160285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-161b1700-7005-4cae-8220-f40abc01efd6 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:00.161472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-161b1700-7005-4cae-8220-f40abc01efd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:00.163019 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:00.181161 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8a7739-e630-425b-8f7d-4afb8bd22ba3 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:00.218162 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1408ca7c-75c7-43fc-81ae-ec14d63dc349 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:00.219283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1408ca7c-75c7-43fc-81ae-ec14d63dc349 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:00.219807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1408ca7c-75c7-43fc-81ae-ec14d63dc349 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:00.220081 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1408ca7c-75c7-43fc-81ae-ec14d63dc349 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:00.220877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1408ca7c-75c7-43fc-81ae-ec14d63dc349 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:00.222456 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1408ca7c-75c7-43fc-81ae-ec14d63dc349 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:00.231868 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 487/969] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:00 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:06:00.296369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 483/970] 10.222.0.47 () {60 vars in 1141 bytes} [Sun Aug 21 01:06:00 2022] POST /identity/v3/services => generated 242 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:06:00.501171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161b1700-7005-4cae-8220-f40abc01efd6 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:00.506857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e4bc0a-15d4-407c-a347-f8d58df5d980 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:00.507150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e4bc0a-15d4-407c-a347-f8d58df5d980 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:00.507429 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e4bc0a-15d4-407c-a347-f8d58df5d980 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:00.509218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 484/971] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:00.514229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2e7e0b-4005-4fe6-b0da-39bfdc13962b None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:00.514514 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2e7e0b-4005-4fe6-b0da-39bfdc13962b None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:00.514789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2e7e0b-4005-4fe6-b0da-39bfdc13962b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:00.534387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-0e2e7e0b-4005-4fe6-b0da-39bfdc13962b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:00.536072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-0e2e7e0b-4005-4fe6-b0da-39bfdc13962b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:00.610704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161b1700-7005-4cae-8220-f40abc01efd6 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:00.614387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 485/972] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:00 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:06:00.633833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:00.648363 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b5c118-bd1b-4f71-bdcc-75bdb9792e7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:00.649058 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b5c118-bd1b-4f71-bdcc-75bdb9792e7c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:00.649339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b5c118-bd1b-4f71-bdcc-75bdb9792e7c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:00.649614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b5c118-bd1b-4f71-bdcc-75bdb9792e7c None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:00.650192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b5c118-bd1b-4f71-bdcc-75bdb9792e7c None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:00.651615 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b5c118-bd1b-4f71-bdcc-75bdb9792e7c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-14b5c118-bd1b-4f71-bdcc-75bdb9792e7c None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:00.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:00.661432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:00.661432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:00.661432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:00.661432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:00.661432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:00.661432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:00.661432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:06:00.661432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:06:00.661432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:06:00.661432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:06:00.661432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:00.661432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:00.661432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:00.661432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:00.661432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:00.661432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:00.662308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:00.662308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:00.662308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:00.662308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:00.662308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:00.662308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:00.662308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:00.662308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:00.662308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:00.662308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:00.662308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:06:00.662308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:06:00.662308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 21 01:06:00.662308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:00.664969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 488/973] 10.222.0.47 () {56 vars in 1126 bytes} [Sun Aug 21 01:06:00 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:00.667338 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2e7e0b-4005-4fe6-b0da-39bfdc13962b None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:00.737716 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:00.770489 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2e7e0b-4005-4fe6-b0da-39bfdc13962b None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:00.805110 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8248fd-b4f9-430f-8101-b7e78cc50d1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:00.805715 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8248fd-b4f9-430f-8101-b7e78cc50d1b None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:00.805994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8248fd-b4f9-430f-8101-b7e78cc50d1b None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:00.806263 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8248fd-b4f9-430f-8101-b7e78cc50d1b None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:00.806885 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8248fd-b4f9-430f-8101-b7e78cc50d1b None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:00.807187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8248fd-b4f9-430f-8101-b7e78cc50d1b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:00.808574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8248fd-b4f9-430f-8101-b7e78cc50d1b None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:00.814580 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 486/974] 10.222.0.47 () {56 vars in 1122 bytes} [Sun Aug 21 01:06:00 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:00.820335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b5c118-bd1b-4f71-bdcc-75bdb9792e7c None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:00.885713 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8248fd-b4f9-430f-8101-b7e78cc50d1b None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:00.927676 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b5c118-bd1b-4f71-bdcc-75bdb9792e7c None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:00.964114 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac9b1d8-7b71-47b6-a1b1-ddad80cc21e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:00.964841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac9b1d8-7b71-47b6-a1b1-ddad80cc21e0 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:00.965213 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac9b1d8-7b71-47b6-a1b1-ddad80cc21e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:00.965477 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac9b1d8-7b71-47b6-a1b1-ddad80cc21e0 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:00.966104 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac9b1d8-7b71-47b6-a1b1-ddad80cc21e0 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:00.966409 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac9b1d8-7b71-47b6-a1b1-ddad80cc21e0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:00.967759 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac9b1d8-7b71-47b6-a1b1-ddad80cc21e0 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:00.973224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 489/975] 10.222.0.47 () {56 vars in 1122 bytes} [Sun Aug 21 01:06:00 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:00.979100 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac9b1d8-7b71-47b6-a1b1-ddad80cc21e0 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:00.993385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8248fd-b4f9-430f-8101-b7e78cc50d1b None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:01.029528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e110215-9f62-4113-ae10-ab105dedb2eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:01.030409 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e110215-9f62-4113-ae10-ab105dedb2eb None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:01.030676 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e110215-9f62-4113-ae10-ab105dedb2eb None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:01.030941 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e110215-9f62-4113-ae10-ab105dedb2eb None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:01.032877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:01.036091 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:01.041466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e110215-9f62-4113-ae10-ab105dedb2eb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:06:01.047266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-6e110215-9f62-4113-ae10-ab105dedb2eb None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 21 01:06:01.047266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:01.047266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:01.047266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:01.047266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:01.047266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:01.047266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:01.047266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:01.047266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:01.047266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:01.047266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:01.047266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:01.047266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:06:01.047266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:06:01.047266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:06:01.047266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:06:01.048390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:06:01.048390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:06:01.048390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:06:01.048390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:06:01.048390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:01.048390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:01.048390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:01.048390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:01.048390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:01.048390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:01.048390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:01.048390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:01.048390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:01.048390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:01.048390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:01.048390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:01.049272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:01.049272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:01.049272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:01.049272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:01.049272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:06:01.049272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:06:01.049272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:06:01.049272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:06:01.049272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:06:01.049272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:06:01.049272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 21 01:06:01.049272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:01.049272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 487/976] 10.222.0.47 () {56 vars in 1120 bytes} [Sun Aug 21 01:06:00 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:06:01.053804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e110215-9f62-4113-ae10-ab105dedb2eb None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:01.092968 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac9b1d8-7b71-47b6-a1b1-ddad80cc21e0 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:01.129767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627eaf7a-8980-4571-b246-e10f0b8ef82e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:01.130376 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627eaf7a-8980-4571-b246-e10f0b8ef82e None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:01.130652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627eaf7a-8980-4571-b246-e10f0b8ef82e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:01.131039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627eaf7a-8980-4571-b246-e10f0b8ef82e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:01.131739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627eaf7a-8980-4571-b246-e10f0b8ef82e None admin] RBAC: Adding query filter params (interface=admin, service_id=44a81662972148fc9d3d68ee5d8c93c9, region_id=RegionOne) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:01.132044 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627eaf7a-8980-4571-b246-e10f0b8ef82e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:01.133562 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627eaf7a-8980-4571-b246-e10f0b8ef82e None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:01.144032 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 490/977] 10.222.0.47 () {56 vars in 1249 bytes} [Sun Aug 21 01:06:00 2022] GET /identity/v3/endpoints?service_id=44a81662972148fc9d3d68ee5d8c93c9&interface=admin®ion_id=RegionOne => generated 193 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:01.144985 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627eaf7a-8980-4571-b246-e10f0b8ef82e None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:01.165436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e110215-9f62-4113-ae10-ab105dedb2eb None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:01.204377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c539190c-1d9b-40d9-92fc-6e25486a79dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:01.205339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c539190c-1d9b-40d9-92fc-6e25486a79dc None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:01.205822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c539190c-1d9b-40d9-92fc-6e25486a79dc None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:01.206247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c539190c-1d9b-40d9-92fc-6e25486a79dc None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:01.207048 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c539190c-1d9b-40d9-92fc-6e25486a79dc None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:01.207520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c539190c-1d9b-40d9-92fc-6e25486a79dc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:01.209045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c539190c-1d9b-40d9-92fc-6e25486a79dc None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:01.217427 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 488/978] 10.222.0.47 () {56 vars in 1115 bytes} [Sun Aug 21 01:06:01 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:01.223559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c539190c-1d9b-40d9-92fc-6e25486a79dc None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:01.256250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627eaf7a-8980-4571-b246-e10f0b8ef82e None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:01.274452 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:01.286757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:01.295800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb87949-c24b-4ca0-ae93-9664929229cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:01.296430 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb87949-c24b-4ca0-ae93-9664929229cf None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:01.296765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb87949-c24b-4ca0-ae93-9664929229cf None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:01.297033 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb87949-c24b-4ca0-ae93-9664929229cf None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:01.297653 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb87949-c24b-4ca0-ae93-9664929229cf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:01.299004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb87949-c24b-4ca0-ae93-9664929229cf None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:01.312761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 491/979] 10.222.0.47 () {56 vars in 1087 bytes} [Sun Aug 21 01:06:01 2022] GET /identity/v3/services => generated 1727 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:06:01.334116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c539190c-1d9b-40d9-92fc-6e25486a79dc None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:01.379085 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7348ca75-65ba-4157-9285-417e8d4932fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:01.379738 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7348ca75-65ba-4157-9285-417e8d4932fc None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:01.380014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7348ca75-65ba-4157-9285-417e8d4932fc None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:01.380314 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7348ca75-65ba-4157-9285-417e8d4932fc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:01.394525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7348ca75-65ba-4157-9285-417e8d4932fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:01.395942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7348ca75-65ba-4157-9285-417e8d4932fc None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:01.411722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-7348ca75-65ba-4157-9285-417e8d4932fc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:01.411722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:01.411722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:01.411722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:01.411722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:01.411722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:01.411722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:01.411722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:01.411722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:01.411722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:01.411722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:01.411722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:01.411722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:06:01.411722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:06:01.411722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:06:01.411722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:01.418297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:06:01.419468 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:06:01.419468 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:01.419468 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:01.419468 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 489/980] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:06:01.421453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb87949-c24b-4ca0-ae93-9664929229cf None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:01.530964 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb87949-c24b-4ca0-ae93-9664929229cf None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:01.565219 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127e2450-3c2d-4013-80eb-b96b86dd6ec1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:01.565820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127e2450-3c2d-4013-80eb-b96b86dd6ec1 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:01.566097 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127e2450-3c2d-4013-80eb-b96b86dd6ec1 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:01.566367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127e2450-3c2d-4013-80eb-b96b86dd6ec1 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:01.567027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127e2450-3c2d-4013-80eb-b96b86dd6ec1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:01.567329 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127e2450-3c2d-4013-80eb-b96b86dd6ec1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:01.568702 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127e2450-3c2d-4013-80eb-b96b86dd6ec1 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:01.585857 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 492/981] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:06:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:01.587295 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7348ca75-65ba-4157-9285-417e8d4932fc None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:01.690138 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7348ca75-65ba-4157-9285-417e8d4932fc None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:01.698911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:01.724783 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9157f6-b7a2-4a63-80c3-dd7263f0f2e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:01.725408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9157f6-b7a2-4a63-80c3-dd7263f0f2e0 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:01.725679 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9157f6-b7a2-4a63-80c3-dd7263f0f2e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:01.725943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9157f6-b7a2-4a63-80c3-dd7263f0f2e0 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:01.733559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9157f6-b7a2-4a63-80c3-dd7263f0f2e0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:01.734996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9157f6-b7a2-4a63-80c3-dd7263f0f2e0 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:01.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-1d9157f6-b7a2-4a63-80c3-dd7263f0f2e0 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 21 01:06:01.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:01.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:01.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:01.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:01.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:01.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:01.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:01.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:01.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:01.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:01.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:01.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:06:01.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:06:01.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:06:01.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:01.748135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:01.749221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:01.749221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:01.749221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:06:01.749221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:06:01.749221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:06:01.749221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:06:01.749221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:06:01.749221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:06:01.749221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 21 01:06:01.749221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:01.756236 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 490/982] 10.222.0.47 () {56 vars in 1117 bytes} [Sun Aug 21 01:06:01 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:01.757795 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127e2450-3c2d-4013-80eb-b96b86dd6ec1 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:01.794286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:01.860506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127e2450-3c2d-4013-80eb-b96b86dd6ec1 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:01.894745 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1ce15b-9c8c-4818-a040-c66e6a330251 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:01.895338 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1ce15b-9c8c-4818-a040-c66e6a330251 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:01.895610 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1ce15b-9c8c-4818-a040-c66e6a330251 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:01.895874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1ce15b-9c8c-4818-a040-c66e6a330251 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:01.896612 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1ce15b-9c8c-4818-a040-c66e6a330251 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:01.896908 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1ce15b-9c8c-4818-a040-c66e6a330251 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:01.898297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1ce15b-9c8c-4818-a040-c66e6a330251 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:01.915465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 493/983] 10.222.0.47 () {56 vars in 1149 bytes} [Sun Aug 21 01:06:01 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:01.920952 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9157f6-b7a2-4a63-80c3-dd7263f0f2e0 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:02.022470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9157f6-b7a2-4a63-80c3-dd7263f0f2e0 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:02.056815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6482f6-8296-4995-8887-a36ce57a341c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:02.057414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6482f6-8296-4995-8887-a36ce57a341c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:02.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6482f6-8296-4995-8887-a36ce57a341c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:02.057951 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6482f6-8296-4995-8887-a36ce57a341c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:02.092324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:02.126609 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6482f6-8296-4995-8887-a36ce57a341c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:02.128447 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6482f6-8296-4995-8887-a36ce57a341c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:02.229060 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d19db0-d081-422a-81fa-48ba381ca74f None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:02.229493 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d19db0-d081-422a-81fa-48ba381ca74f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:02.229614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d19db0-d081-422a-81fa-48ba381ca74f None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:02.231071 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 494/984] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:06:02.233342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 491/985] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:06:01 2022] GET /identity/v3/auth/tokens => generated 1570 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:06:02.233460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d19db0-d081-422a-81fa-48ba381ca74f None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:02.236638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95855426-aaef-49a8-a56f-2b7389361517 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:02.236919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95855426-aaef-49a8-a56f-2b7389361517 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:02.237213 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95855426-aaef-49a8-a56f-2b7389361517 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:02.254453 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-95855426-aaef-49a8-a56f-2b7389361517 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:02.255633 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-95855426-aaef-49a8-a56f-2b7389361517 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:02.334995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:02.335977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:02.344146 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 492/986] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:02 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:02.352649 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d19db0-d081-422a-81fa-48ba381ca74f None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:02.386515 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15653c6-52ec-49a5-902e-0f1732d577aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:02.387149 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15653c6-52ec-49a5-902e-0f1732d577aa None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:02.387423 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15653c6-52ec-49a5-902e-0f1732d577aa None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:02.387716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15653c6-52ec-49a5-902e-0f1732d577aa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:02.398758 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15653c6-52ec-49a5-902e-0f1732d577aa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:02.400258 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15653c6-52ec-49a5-902e-0f1732d577aa None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:02.416670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-a15653c6-52ec-49a5-902e-0f1732d577aa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:02.416670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:02.416670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:02.416670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:02.416670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:02.416670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:02.416670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:02.416670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:02.416670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:02.416670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:02.416670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:02.416670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:02.416670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:06:02.416670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:06:02.416670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:06:02.416670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:02.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:06:02.418652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:06:02.418652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:02.418652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:02.419392 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 495/987] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:02.424287 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95855426-aaef-49a8-a56f-2b7389361517 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:02.523839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95855426-aaef-49a8-a56f-2b7389361517 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:02.556720 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ef9eaa-4842-4e6a-8780-6b65a52cbf8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:02.557313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ef9eaa-4842-4e6a-8780-6b65a52cbf8a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:02.557581 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ef9eaa-4842-4e6a-8780-6b65a52cbf8a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:02.557846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ef9eaa-4842-4e6a-8780-6b65a52cbf8a None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:02.558471 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ef9eaa-4842-4e6a-8780-6b65a52cbf8a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:02.558761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ef9eaa-4842-4e6a-8780-6b65a52cbf8a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:02.560117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ef9eaa-4842-4e6a-8780-6b65a52cbf8a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:02.572131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 493/988] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:06:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:02.577526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15653c6-52ec-49a5-902e-0f1732d577aa None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:02.611602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ef9eaa-4842-4e6a-8780-6b65a52cbf8a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:02.673310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:02.696295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15653c6-52ec-49a5-902e-0f1732d577aa None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:02.718058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ef9eaa-4842-4e6a-8780-6b65a52cbf8a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:02.734607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2b47bd-fd8c-4a5d-b923-28bdf9064142 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:02.735480 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2b47bd-fd8c-4a5d-b923-28bdf9064142 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:02.735907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2b47bd-fd8c-4a5d-b923-28bdf9064142 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:02.736353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2b47bd-fd8c-4a5d-b923-28bdf9064142 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:02.743501 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2b47bd-fd8c-4a5d-b923-28bdf9064142 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:02.745160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2b47bd-fd8c-4a5d-b923-28bdf9064142 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:02.753565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b68574-ead4-4040-8f55-b8a5713aa11e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:02.754197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b68574-ead4-4040-8f55-b8a5713aa11e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:02.754472 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b68574-ead4-4040-8f55-b8a5713aa11e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:02.754741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b68574-ead4-4040-8f55-b8a5713aa11e None admin] PATH_INFO: `/v3/services/compute` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:02.755317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b68574-ead4-4040-8f55-b8a5713aa11e None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:02.756759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b68574-ead4-4040-8f55-b8a5713aa11e None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:02.758162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-9a2b47bd-fd8c-4a5d-b923-28bdf9064142 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:06:02.758162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:02.758162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:02.758162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:02.758162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:02.758162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:02.758162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:02.758162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:02.758162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:02.758162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:02.758162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:02.758162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:02.758162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:06:02.758162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:06:02.758162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:02.759242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:02.760346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:06:02.760346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:06:02.760346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:06:02.760346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:06:02.760346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:06:02.760346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:06:02.760346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:06:02.760346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:02.761467 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 496/989] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:06:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:02.767555 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2b47bd-fd8c-4a5d-b923-28bdf9064142 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-54b68574-ead4-4040-8f55-b8a5713aa11e None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:02.768027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:02.769117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:02.769117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:02.769117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:02.769117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:02.769117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:02.769117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:02.769117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:06:02.769117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:06:02.769117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:06:02.769117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:06:02.769117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:02.769117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:02.769117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:02.769117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:02.769117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:02.769117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:02.769978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:02.769978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:02.769978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:02.769978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:02.769978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:02.769978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:02.769978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:02.769978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:02.769978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:02.769978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:02.769978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:06:02.769978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:06:02.769978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 21 01:06:02.769978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:02.773922 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 494/990] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:06:02 2022] GET /identity/v3/services/compute => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:06:02.774655 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b68574-ead4-4040-8f55-b8a5713aa11e None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:02.779475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:02.878164 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2b47bd-fd8c-4a5d-b923-28bdf9064142 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:02.885714 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b68574-ead4-4040-8f55-b8a5713aa11e None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:02.911387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:02.915012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b76b50-4319-4aa6-a2bb-4c54a0389261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:02.915856 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b76b50-4319-4aa6-a2bb-4c54a0389261 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:02.916366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b76b50-4319-4aa6-a2bb-4c54a0389261 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:02.916817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b76b50-4319-4aa6-a2bb-4c54a0389261 None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:02.917708 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b76b50-4319-4aa6-a2bb-4c54a0389261 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:02.918192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b76b50-4319-4aa6-a2bb-4c54a0389261 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:02.919764 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b76b50-4319-4aa6-a2bb-4c54a0389261 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:02.925309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e19e6fc-0856-4225-8b8f-fc2a0ec2ae83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:02.925917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e19e6fc-0856-4225-8b8f-fc2a0ec2ae83 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:02.926192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e19e6fc-0856-4225-8b8f-fc2a0ec2ae83 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:02.926460 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e19e6fc-0856-4225-8b8f-fc2a0ec2ae83 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:02.927089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e19e6fc-0856-4225-8b8f-fc2a0ec2ae83 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:02.927386 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e19e6fc-0856-4225-8b8f-fc2a0ec2ae83 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:02.928760 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e19e6fc-0856-4225-8b8f-fc2a0ec2ae83 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:02.940157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 495/991] 10.222.0.47 () {56 vars in 1112 bytes} [Sun Aug 21 01:06:02 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:02.940326 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 497/992] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:06:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:02.941329 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e19e6fc-0856-4225-8b8f-fc2a0ec2ae83 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:02.942072 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b76b50-4319-4aa6-a2bb-4c54a0389261 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:03.011505 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:03.050233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b76b50-4319-4aa6-a2bb-4c54a0389261 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:03.050668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e19e6fc-0856-4225-8b8f-fc2a0ec2ae83 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:03.085878 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba93c81f-e557-45f3-8328-b37a2b897a0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:03.086574 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba93c81f-e557-45f3-8328-b37a2b897a0a None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:03.086849 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba93c81f-e557-45f3-8328-b37a2b897a0a None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:03.087119 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba93c81f-e557-45f3-8328-b37a2b897a0a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:03.087848 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba93c81f-e557-45f3-8328-b37a2b897a0a None admin] RBAC: Adding query filter params (role.id=830cf269498649038ec8a27a3712fc60, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=d7c65195ff8d42958fa54b5f083bf8da) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:03.088051 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c0536a-2f73-4a88-ac54-3e309ace4c53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:03.088218 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba93c81f-e557-45f3-8328-b37a2b897a0a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:03.088945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c0536a-2f73-4a88-ac54-3e309ace4c53 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:03.089360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c0536a-2f73-4a88-ac54-3e309ace4c53 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:03.089577 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba93c81f-e557-45f3-8328-b37a2b897a0a None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:03.089921 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c0536a-2f73-4a88-ac54-3e309ace4c53 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:03.090691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c0536a-2f73-4a88-ac54-3e309ace4c53 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:03.091149 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c0536a-2f73-4a88-ac54-3e309ace4c53 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:03.092825 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c0536a-2f73-4a88-ac54-3e309ace4c53 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:03.096140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 498/993] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:06:02 2022] GET /identity/v3/role_assignments?user.id=d7c65195ff8d42958fa54b5f083bf8da&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=830cf269498649038ec8a27a3712fc60 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:03.103232 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 496/994] 10.222.0.47 () {56 vars in 1112 bytes} [Sun Aug 21 01:06:02 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:03.104840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba93c81f-e557-45f3-8328-b37a2b897a0a None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:03.217739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba93c81f-e557-45f3-8328-b37a2b897a0a None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:03.240452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d022f818-ef12-4a57-88c5-e93265a26bd5 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:03.240738 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d022f818-ef12-4a57-88c5-e93265a26bd5 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:03.241009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d022f818-ef12-4a57-88c5-e93265a26bd5 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:03.243582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 497/995] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:06:03.248255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4778b845-d530-4ac5-9a4b-c7690742be31 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:03.248695 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4778b845-d530-4ac5-9a4b-c7690742be31 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:03.248969 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4778b845-d530-4ac5-9a4b-c7690742be31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:03.250595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:03.256936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d054b2-258c-4de1-b027-4e5dc5e917de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:03.257544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d054b2-258c-4de1-b027-4e5dc5e917de None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:03.257821 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d054b2-258c-4de1-b027-4e5dc5e917de None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:03.258091 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d054b2-258c-4de1-b027-4e5dc5e917de None admin] PATH_INFO: `/v3/endpoints` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:03.259666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d054b2-258c-4de1-b027-4e5dc5e917de None admin] RBAC: Adding query filter params (interface=public, service_id=68a24b970ffb4bd994f1cc618445d952, region_id=RegionOne) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:03.259980 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d054b2-258c-4de1-b027-4e5dc5e917de None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:03.261367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d054b2-258c-4de1-b027-4e5dc5e917de None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:03.267738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 499/996] 10.222.0.47 () {56 vars in 1251 bytes} [Sun Aug 21 01:06:03 2022] GET /identity/v3/endpoints?service_id=68a24b970ffb4bd994f1cc618445d952&interface=public®ion_id=RegionOne => generated 194 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:03.271391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-4778b845-d530-4ac5-9a4b-c7690742be31 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:03.272707 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-4778b845-d530-4ac5-9a4b-c7690742be31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:03.273368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d054b2-258c-4de1-b027-4e5dc5e917de None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:03.323356 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:03.349033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 498/997] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:03 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:03.398276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d054b2-258c-4de1-b027-4e5dc5e917de None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:03.433518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163b984b-ecbd-4720-99af-5fd36c5e865e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:03.434472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163b984b-ecbd-4720-99af-5fd36c5e865e None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:03.434966 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163b984b-ecbd-4720-99af-5fd36c5e865e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:03.435429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163b984b-ecbd-4720-99af-5fd36c5e865e None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:03.436264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163b984b-ecbd-4720-99af-5fd36c5e865e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:03.438354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163b984b-ecbd-4720-99af-5fd36c5e865e None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:03.452143 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 500/998] 10.222.0.47 () {56 vars in 1087 bytes} [Sun Aug 21 01:06:03 2022] GET /identity/v3/services => generated 1727 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:06:03.614799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4778b845-d530-4ac5-9a4b-c7690742be31 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:03.716477 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4778b845-d530-4ac5-9a4b-c7690742be31 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:03.751674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93ca67b-cf52-483f-88f6-7b7dd214bd89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:03.752329 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93ca67b-cf52-483f-88f6-7b7dd214bd89 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:03.752599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93ca67b-cf52-483f-88f6-7b7dd214bd89 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:03.752860 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93ca67b-cf52-483f-88f6-7b7dd214bd89 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:03.753433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93ca67b-cf52-483f-88f6-7b7dd214bd89 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:03.754778 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93ca67b-cf52-483f-88f6-7b7dd214bd89 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-c93ca67b-cf52-483f-88f6-7b7dd214bd89 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:03.764017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:03.765229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:03.765229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:03.765229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:03.765229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:03.765229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:03.765229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:03.765229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:06:03.765229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:06:03.765229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:06:03.765229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:06:03.765229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:03.765229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:03.765229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:03.765229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:03.765229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:03.765229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:03.766144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:03.766144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:03.766144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:03.766144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:03.766144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:03.766144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:03.766144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:03.766144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:03.766144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:03.766144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:03.766144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:06:03.766144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:06:03.766144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 21 01:06:03.766144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:03.767432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 499/999] 10.222.0.47 () {56 vars in 1126 bytes} [Sun Aug 21 01:06:03 2022] GET /identity/v3/services/object-store => generated 93 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:03.771659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163b984b-ecbd-4720-99af-5fd36c5e865e None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:03.830230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:03.871850 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163b984b-ecbd-4720-99af-5fd36c5e865e None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:03.905298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428e55a2-e631-465a-a6b7-cdbedb04726d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:03.906552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428e55a2-e631-465a-a6b7-cdbedb04726d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:03.906775 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428e55a2-e631-465a-a6b7-cdbedb04726d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:03.906988 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428e55a2-e631-465a-a6b7-cdbedb04726d None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:03.907599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428e55a2-e631-465a-a6b7-cdbedb04726d None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:03.907894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428e55a2-e631-465a-a6b7-cdbedb04726d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:03.909263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428e55a2-e631-465a-a6b7-cdbedb04726d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:03.914007 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 501/1000] 10.222.0.47 () {56 vars in 1122 bytes} [Sun Aug 21 01:06:03 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:03.919660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93ca67b-cf52-483f-88f6-7b7dd214bd89 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:03.969319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:04.025118 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93ca67b-cf52-483f-88f6-7b7dd214bd89 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:04.060196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5633b530-23df-46a7-abb1-5b57cf6d9dd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:04.060795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5633b530-23df-46a7-abb1-5b57cf6d9dd7 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:04.061066 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5633b530-23df-46a7-abb1-5b57cf6d9dd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:04.061329 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5633b530-23df-46a7-abb1-5b57cf6d9dd7 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:04.061985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5633b530-23df-46a7-abb1-5b57cf6d9dd7 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:04.062279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5633b530-23df-46a7-abb1-5b57cf6d9dd7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:04.063624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5633b530-23df-46a7-abb1-5b57cf6d9dd7 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:04.073417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 500/1001] 10.222.0.47 () {56 vars in 1122 bytes} [Sun Aug 21 01:06:03 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:04.074551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428e55a2-e631-465a-a6b7-cdbedb04726d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:04.178171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428e55a2-e631-465a-a6b7-cdbedb04726d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:04.200260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:04.212669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa45d4ec-0aef-474e-aead-b558499be4f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:04.213366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa45d4ec-0aef-474e-aead-b558499be4f0 None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:04.213644 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa45d4ec-0aef-474e-aead-b558499be4f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:04.213909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa45d4ec-0aef-474e-aead-b558499be4f0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:04.214558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa45d4ec-0aef-474e-aead-b558499be4f0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:04.215895 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa45d4ec-0aef-474e-aead-b558499be4f0 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:04.280461 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 502/1002] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:06:04 2022] POST /identity/v3/endpoints => generated 325 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:06:05.053339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d4d668-c248-4086-93ba-04441cd6cee2 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:05.053976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d4d668-c248-4086-93ba-04441cd6cee2 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:05.053976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d4d668-c248-4086-93ba-04441cd6cee2 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:05.055420 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 501/1003] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:05.060637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185226d6-d06b-4d1f-9c28-dddee85b8778 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:05.061076 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185226d6-d06b-4d1f-9c28-dddee85b8778 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:05.061233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185226d6-d06b-4d1f-9c28-dddee85b8778 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:05.080135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-185226d6-d06b-4d1f-9c28-dddee85b8778 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:05.081319 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-185226d6-d06b-4d1f-9c28-dddee85b8778 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:05.157286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 503/1004] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:05 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:05.343607 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2a9d99-0218-49b3-a601-a60dca6d7199 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:05.343799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2a9d99-0218-49b3-a601-a60dca6d7199 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:05.345816 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2a9d99-0218-49b3-a601-a60dca6d7199 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:05.345816 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 502/1005] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:05.351696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34896881-1f4f-462f-bc6b-95e233ccec3b None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:05.354703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34896881-1f4f-462f-bc6b-95e233ccec3b None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:05.354703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34896881-1f4f-462f-bc6b-95e233ccec3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:05.391761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-34896881-1f4f-462f-bc6b-95e233ccec3b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:05.392968 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-34896881-1f4f-462f-bc6b-95e233ccec3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:05.431949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2a9d99-0218-49b3-a601-a60dca6d7199 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:05.464511 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 504/1006] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:05 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:05.540535 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2a9d99-0218-49b3-a601-a60dca6d7199 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:05.549161 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:05.573441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e453a5f-b31c-489d-8ee8-30cc44a0bc32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:05.574061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e453a5f-b31c-489d-8ee8-30cc44a0bc32 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:05.574334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e453a5f-b31c-489d-8ee8-30cc44a0bc32 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:05.574599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e453a5f-b31c-489d-8ee8-30cc44a0bc32 None admin] PATH_INFO: `/v3/services/image` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:05.575164 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e453a5f-b31c-489d-8ee8-30cc44a0bc32 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:05.576623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e453a5f-b31c-489d-8ee8-30cc44a0bc32 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-1e453a5f-b31c-489d-8ee8-30cc44a0bc32 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:05.585725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:05.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:05.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:05.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:05.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:05.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:05.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:05.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:06:05.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:06:05.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:06:05.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:06:05.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:05.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:05.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:05.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:05.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:05.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:05.588214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:05.588214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:05.588214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:05.588214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:05.588214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:05.588214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:05.588214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:05.588214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:05.588214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:05.588214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:05.588214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:06:05.588214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:06:05.588214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 21 01:06:05.588214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:05.588214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 503/1007] 10.222.0.47 () {56 vars in 1105 bytes} [Sun Aug 21 01:06:05 2022] GET /identity/v3/services/image => generated 86 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:06:05.592790 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34896881-1f4f-462f-bc6b-95e233ccec3b None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:05.634364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:05.696370 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34896881-1f4f-462f-bc6b-95e233ccec3b None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:05.733467 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea86ee4-6d73-4ac9-baa8-0d6dfbbc997a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:05.734167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea86ee4-6d73-4ac9-baa8-0d6dfbbc997a None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:05.734444 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea86ee4-6d73-4ac9-baa8-0d6dfbbc997a None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:05.734715 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea86ee4-6d73-4ac9-baa8-0d6dfbbc997a None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:05.735342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea86ee4-6d73-4ac9-baa8-0d6dfbbc997a None admin] RBAC: Adding query filter params (name=image) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:05.735645 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea86ee4-6d73-4ac9-baa8-0d6dfbbc997a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:05.737028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea86ee4-6d73-4ac9-baa8-0d6dfbbc997a None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:05.739525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e453a5f-b31c-489d-8ee8-30cc44a0bc32 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:05.742350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 505/1008] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:05 2022] GET /identity/v3/services?name=image => generated 122 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:05.747801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea86ee4-6d73-4ac9-baa8-0d6dfbbc997a None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:05.851756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e453a5f-b31c-489d-8ee8-30cc44a0bc32 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:05.855052 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea86ee4-6d73-4ac9-baa8-0d6dfbbc997a None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:05.856720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:05.888585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2790cd9-9d05-4210-9a33-46a63650fdf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:05.889236 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2790cd9-9d05-4210-9a33-46a63650fdf6 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:05.889417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8800e84-b4e0-4d32-b7f1-49c262d2ff42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:05.889533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2790cd9-9d05-4210-9a33-46a63650fdf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:05.889772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2790cd9-9d05-4210-9a33-46a63650fdf6 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:05.890555 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8800e84-b4e0-4d32-b7f1-49c262d2ff42 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:05.890793 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8800e84-b4e0-4d32-b7f1-49c262d2ff42 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:05.890901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2790cd9-9d05-4210-9a33-46a63650fdf6 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:05.891522 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8800e84-b4e0-4d32-b7f1-49c262d2ff42 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:05.892275 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2790cd9-9d05-4210-9a33-46a63650fdf6 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:05.892516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8800e84-b4e0-4d32-b7f1-49c262d2ff42 None admin] RBAC: Adding query filter params (type=image) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:05.893506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8800e84-b4e0-4d32-b7f1-49c262d2ff42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:05.894843 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8800e84-b4e0-4d32-b7f1-49c262d2ff42 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:05.899703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 506/1009] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:05 2022] GET /identity/v3/services?type=image => generated 122 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-d2790cd9-9d05-4210-9a33-46a63650fdf6 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:05.901381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:05.902377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:05.902377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:05.902377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:05.902377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:05.902377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:05.902377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:05.902377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:06:05.902377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:06:05.902377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:06:05.902377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:06:05.902377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:05.902377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:05.902377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:05.902377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:05.902377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:05.902377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:05.903272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:05.903272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:05.903272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:05.903272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:05.903272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:05.903272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:05.903272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:05.903272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:05.903272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:05.903272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:05.903272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:06:05.903272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:06:05.903272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 21 01:06:05.903272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:05.903272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 504/1010] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:06:05 2022] GET /identity/v3/services/compute => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:06:05.909891 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8800e84-b4e0-4d32-b7f1-49c262d2ff42 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:06.019733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8800e84-b4e0-4d32-b7f1-49c262d2ff42 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:06.059411 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1aacaf8-dea1-4f10-ac5a-27f0ed425b03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:06.060019 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1aacaf8-dea1-4f10-ac5a-27f0ed425b03 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:06.060434 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1aacaf8-dea1-4f10-ac5a-27f0ed425b03 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:06.060710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1aacaf8-dea1-4f10-ac5a-27f0ed425b03 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:06.061343 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1aacaf8-dea1-4f10-ac5a-27f0ed425b03 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:06.061643 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1aacaf8-dea1-4f10-ac5a-27f0ed425b03 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:06.063522 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1aacaf8-dea1-4f10-ac5a-27f0ed425b03 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:06.068920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 507/1011] 10.222.0.47 () {56 vars in 1112 bytes} [Sun Aug 21 01:06:05 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:06.074391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2790cd9-9d05-4210-9a33-46a63650fdf6 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:06.112716 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:06.177576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2790cd9-9d05-4210-9a33-46a63650fdf6 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:06.201409 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfad3991-c310-432a-855c-accee3a61311 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:06.201711 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfad3991-c310-432a-855c-accee3a61311 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:06.201963 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfad3991-c310-432a-855c-accee3a61311 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:06.204279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 508/1012] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:06:06.209535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bed980-c111-460f-8cc3-c48fb7325999 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:06.209820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bed980-c111-460f-8cc3-c48fb7325999 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:06.210088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bed980-c111-460f-8cc3-c48fb7325999 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:06.212778 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa1291a-96f1-4afc-9378-dded5f8b91d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:06.213382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa1291a-96f1-4afc-9378-dded5f8b91d8 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:06.213654 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa1291a-96f1-4afc-9378-dded5f8b91d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:06.213919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa1291a-96f1-4afc-9378-dded5f8b91d8 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:06.214541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa1291a-96f1-4afc-9378-dded5f8b91d8 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:06.214836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa1291a-96f1-4afc-9378-dded5f8b91d8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:06.216204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa1291a-96f1-4afc-9378-dded5f8b91d8 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:06.226230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 505/1013] 10.222.0.47 () {56 vars in 1112 bytes} [Sun Aug 21 01:06:06 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:06.227707 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa1291a-96f1-4afc-9378-dded5f8b91d8 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:06.230081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-e2bed980-c111-460f-8cc3-c48fb7325999 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:06.231269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e2bed980-c111-460f-8cc3-c48fb7325999 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:06.257180 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:06.303571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 509/1014] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:06 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:06:06.330395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:06.334405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa1291a-96f1-4afc-9378-dded5f8b91d8 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:06.377568 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215652b7-ed71-405a-b477-c08e0ef7b294 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:06.378537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215652b7-ed71-405a-b477-c08e0ef7b294 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:06.379023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215652b7-ed71-405a-b477-c08e0ef7b294 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:06.379501 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215652b7-ed71-405a-b477-c08e0ef7b294 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:06.380426 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215652b7-ed71-405a-b477-c08e0ef7b294 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:06.381990 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215652b7-ed71-405a-b477-c08e0ef7b294 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:06.486207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 506/1015] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:06:06 2022] POST /identity/v3/endpoints => generated 334 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:06:06.574180 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bed980-c111-460f-8cc3-c48fb7325999 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:06.682119 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bed980-c111-460f-8cc3-c48fb7325999 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:06.723655 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db9b5b9-f9f1-4302-ad96-f2734d617c8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:06.724335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db9b5b9-f9f1-4302-ad96-f2734d617c8d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:06.724617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db9b5b9-f9f1-4302-ad96-f2734d617c8d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:06.724886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db9b5b9-f9f1-4302-ad96-f2734d617c8d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:06.726830 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db9b5b9-f9f1-4302-ad96-f2734d617c8d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:06.728292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db9b5b9-f9f1-4302-ad96-f2734d617c8d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:06.730034 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 510/1016] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:06.736533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215652b7-ed71-405a-b477-c08e0ef7b294 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:06.839669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215652b7-ed71-405a-b477-c08e0ef7b294 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:06.874422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7639999-568b-4a79-94c2-599af44a2d3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:06.875026 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7639999-568b-4a79-94c2-599af44a2d3a None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:06.875300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7639999-568b-4a79-94c2-599af44a2d3a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:06.875570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7639999-568b-4a79-94c2-599af44a2d3a None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:06.876246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7639999-568b-4a79-94c2-599af44a2d3a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:06.877680 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7639999-568b-4a79-94c2-599af44a2d3a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:07.007182 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 507/1017] 10.222.0.47 () {60 vars in 1141 bytes} [Sun Aug 21 01:06:06 2022] POST /identity/v3/projects => generated 311 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:06:07.835331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd8060b-9259-4d17-a87e-e68d38355fe6 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:07.835945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd8060b-9259-4d17-a87e-e68d38355fe6 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:07.836057 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd8060b-9259-4d17-a87e-e68d38355fe6 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:07.837572 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 511/1018] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:07.842526 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f7202c-dd21-4574-8327-d7710f740a73 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:07.843572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f7202c-dd21-4574-8327-d7710f740a73 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:07.843743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f7202c-dd21-4574-8327-d7710f740a73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:07.861404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-40f7202c-dd21-4574-8327-d7710f740a73 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:07.862590 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-40f7202c-dd21-4574-8327-d7710f740a73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:07.864821 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:07.932549 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 508/1019] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:07 2022] POST /identity/v3/auth/tokens => generated 1873 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:08.213116 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd8060b-9259-4d17-a87e-e68d38355fe6 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:08.240333 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:08.317240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd8060b-9259-4d17-a87e-e68d38355fe6 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:08.361033 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7d54ce-332c-4541-b39d-82e5ff5f34e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:08.361654 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7d54ce-332c-4541-b39d-82e5ff5f34e1 None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:08.361934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7d54ce-332c-4541-b39d-82e5ff5f34e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:08.362207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7d54ce-332c-4541-b39d-82e5ff5f34e1 None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:08.362870 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7d54ce-332c-4541-b39d-82e5ff5f34e1 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:08.364276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7d54ce-332c-4541-b39d-82e5ff5f34e1 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:08.432552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 512/1020] 10.222.0.47 () {60 vars in 1141 bytes} [Sun Aug 21 01:06:08 2022] POST /identity/v3/services => generated 242 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:06:08.486937 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65f48bc-4938-453d-ba54-1960a37316a1 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:08.487255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65f48bc-4938-453d-ba54-1960a37316a1 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:08.487439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65f48bc-4938-453d-ba54-1960a37316a1 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:08.492154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 509/1021] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:08.494837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1a0681-90db-4dd7-8f36-b4be01664712 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:08.495163 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1a0681-90db-4dd7-8f36-b4be01664712 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:08.495410 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1a0681-90db-4dd7-8f36-b4be01664712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:08.516491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-0e1a0681-90db-4dd7-8f36-b4be01664712 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:08.517717 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-0e1a0681-90db-4dd7-8f36-b4be01664712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:08.599986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 513/1022] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:08 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:08.873068 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65f48bc-4938-453d-ba54-1960a37316a1 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:08.932277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c55e12-1887-4edd-a832-42596bab7adb None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:08.932787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c55e12-1887-4edd-a832-42596bab7adb None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:08.932897 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c55e12-1887-4edd-a832-42596bab7adb None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:08.934281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 514/1023] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:06:08.939743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3fc503-53d9-4ede-aab2-6e259e3396e7 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:08.940021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3fc503-53d9-4ede-aab2-6e259e3396e7 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:08.940419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3fc503-53d9-4ede-aab2-6e259e3396e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:08.958769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-fa3fc503-53d9-4ede-aab2-6e259e3396e7 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:08.959939 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-fa3fc503-53d9-4ede-aab2-6e259e3396e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:08.963741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:08.983385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65f48bc-4938-453d-ba54-1960a37316a1 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:09.020557 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12714c3e-23a0-486e-91a5-9752ee34b3f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:09.021179 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12714c3e-23a0-486e-91a5-9752ee34b3f6 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:09.021450 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12714c3e-23a0-486e-91a5-9752ee34b3f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:09.021718 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12714c3e-23a0-486e-91a5-9752ee34b3f6 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:09.028447 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:09.032312 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12714c3e-23a0-486e-91a5-9752ee34b3f6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:06:09.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-12714c3e-23a0-486e-91a5-9752ee34b3f6 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 21 01:06:09.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:09.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:09.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:09.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:09.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:09.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:09.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:09.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:09.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:09.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:09.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:09.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:06:09.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:06:09.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:06:09.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:06:09.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:06:09.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:06:09.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:06:09.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:06:09.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:09.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:09.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:09.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:09.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:09.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:09.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:09.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:09.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:09.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:09.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:09.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:09.047442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:09.047442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:09.047442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:09.047442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:09.047442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:06:09.047442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:06:09.047442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:06:09.047442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:06:09.047442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:06:09.047442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:06:09.047442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 21 01:06:09.047442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:09.047442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 510/1024] 10.222.0.47 () {56 vars in 1120 bytes} [Sun Aug 21 01:06:08 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:09.048200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 515/1025] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:08 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:06:09.048276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12714c3e-23a0-486e-91a5-9752ee34b3f6 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:09.150017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12714c3e-23a0-486e-91a5-9752ee34b3f6 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:09.183502 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e6169a-8784-439d-92c5-3396ad1333ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:09.184116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e6169a-8784-439d-92c5-3396ad1333ce None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:09.184413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e6169a-8784-439d-92c5-3396ad1333ce None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:09.184683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e6169a-8784-439d-92c5-3396ad1333ce None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:09.185338 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e6169a-8784-439d-92c5-3396ad1333ce None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:09.185636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e6169a-8784-439d-92c5-3396ad1333ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:09.187515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e6169a-8784-439d-92c5-3396ad1333ce None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:09.195803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 511/1026] 10.222.0.47 () {56 vars in 1115 bytes} [Sun Aug 21 01:06:09 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:09.201298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3fc503-53d9-4ede-aab2-6e259e3396e7 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:09.301880 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3fc503-53d9-4ede-aab2-6e259e3396e7 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:09.303021 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e6169a-8784-439d-92c5-3396ad1333ce None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:09.339607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035d638e-e37a-4125-9b73-e7a894f6d395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:09.340271 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035d638e-e37a-4125-9b73-e7a894f6d395 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:09.340552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035d638e-e37a-4125-9b73-e7a894f6d395 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:09.340824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035d638e-e37a-4125-9b73-e7a894f6d395 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:09.354109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:09.356971 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035d638e-e37a-4125-9b73-e7a894f6d395 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:09.358977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035d638e-e37a-4125-9b73-e7a894f6d395 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:09.372393 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:09.374322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-035d638e-e37a-4125-9b73-e7a894f6d395 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:09.374322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:09.374322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:09.374322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:09.374322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:09.374322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:09.374322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:09.374322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:09.374322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:09.374322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:09.374322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:09.374322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:09.374322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:06:09.374322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:06:09.374322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:06:09.374322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:09.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:06:09.376266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:06:09.376266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:09.376266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:09.380411 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 516/1027] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:09.381596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035d638e-e37a-4125-9b73-e7a894f6d395 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:09.419834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e6169a-8784-439d-92c5-3396ad1333ce None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:09.457133 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e26eca-008a-4faa-8a8e-3ee83c24a2b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:09.458020 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e26eca-008a-4faa-8a8e-3ee83c24a2b5 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:09.458295 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e26eca-008a-4faa-8a8e-3ee83c24a2b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:09.458563 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e26eca-008a-4faa-8a8e-3ee83c24a2b5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:09.459652 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e26eca-008a-4faa-8a8e-3ee83c24a2b5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:09.461394 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e26eca-008a-4faa-8a8e-3ee83c24a2b5 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:09.462991 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 512/1028] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:09.469175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e26eca-008a-4faa-8a8e-3ee83c24a2b5 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:09.498516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035d638e-e37a-4125-9b73-e7a894f6d395 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:09.537247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2848e5-dc70-45b5-9323-8045a0b9bba7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:09.537875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2848e5-dc70-45b5-9323-8045a0b9bba7 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:09.538152 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2848e5-dc70-45b5-9323-8045a0b9bba7 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:09.538494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2848e5-dc70-45b5-9323-8045a0b9bba7 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:09.539068 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2848e5-dc70-45b5-9323-8045a0b9bba7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:09.539369 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2848e5-dc70-45b5-9323-8045a0b9bba7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:09.540781 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2848e5-dc70-45b5-9323-8045a0b9bba7 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:09.553674 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 517/1029] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:06:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:09.559141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2848e5-dc70-45b5-9323-8045a0b9bba7 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:09.576315 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e26eca-008a-4faa-8a8e-3ee83c24a2b5 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:09.606473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:09.615726 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a9a9d9-6af6-41c4-aff3-9cdd7e05fe52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:09.616366 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a9a9d9-6af6-41c4-aff3-9cdd7e05fe52 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:09.620885 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a9a9d9-6af6-41c4-aff3-9cdd7e05fe52 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:09.620885 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a9a9d9-6af6-41c4-aff3-9cdd7e05fe52 None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:09.620885 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a9a9d9-6af6-41c4-aff3-9cdd7e05fe52 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:09.620885 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a9a9d9-6af6-41c4-aff3-9cdd7e05fe52 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:09.622719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:09.670843 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2848e5-dc70-45b5-9323-8045a0b9bba7 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:09.704366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a351d961-8235-43f4-a44b-5f63d05b2867 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:09.704982 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a351d961-8235-43f4-a44b-5f63d05b2867 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:09.705258 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a351d961-8235-43f4-a44b-5f63d05b2867 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:09.705526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a351d961-8235-43f4-a44b-5f63d05b2867 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:09.712671 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a351d961-8235-43f4-a44b-5f63d05b2867 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:09.714135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a351d961-8235-43f4-a44b-5f63d05b2867 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:09.724138 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 513/1030] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:06:09 2022] POST /identity/v3/users => generated 285 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:06:09.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-a351d961-8235-43f4-a44b-5f63d05b2867 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:06:09.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:09.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:09.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:09.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:09.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:09.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:09.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:09.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:09.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:09.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:09.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:09.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:06:09.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:06:09.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:06:09.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:09.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:09.727805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:09.727805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:09.727805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:06:09.727805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:06:09.727805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:06:09.727805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:06:09.727805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:06:09.727805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:06:09.727805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:06:09.727805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:09.727805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 518/1031] 10.222.0.47 () {56 vars in 1093 bytes} [Sun Aug 21 01:06:09 2022] GET /identity/v3/users/nova => generated 82 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:06:09.733372 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a9a9d9-6af6-41c4-aff3-9cdd7e05fe52 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:09.849047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a9a9d9-6af6-41c4-aff3-9cdd7e05fe52 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:09.888735 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2faf794f-29bb-493b-8d93-ae4db7aa22ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:09.889382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faf794f-29bb-493b-8d93-ae4db7aa22ca None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:09.889645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faf794f-29bb-493b-8d93-ae4db7aa22ca None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:09.889914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faf794f-29bb-493b-8d93-ae4db7aa22ca None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:09.890647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2faf794f-29bb-493b-8d93-ae4db7aa22ca None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:09.890946 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2faf794f-29bb-493b-8d93-ae4db7aa22ca None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:09.892391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2faf794f-29bb-493b-8d93-ae4db7aa22ca None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:09.911388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 514/1032] 10.222.0.47 () {56 vars in 1133 bytes} [Sun Aug 21 01:06:09 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:09.917108 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a351d961-8235-43f4-a44b-5f63d05b2867 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:10.023966 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a351d961-8235-43f4-a44b-5f63d05b2867 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:10.038451 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:10.060001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b257c35-22b0-4120-ac05-0798a8ccf539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:10.060636 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b257c35-22b0-4120-ac05-0798a8ccf539 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:10.060913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b257c35-22b0-4120-ac05-0798a8ccf539 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:10.061185 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b257c35-22b0-4120-ac05-0798a8ccf539 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:10.133740 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b257c35-22b0-4120-ac05-0798a8ccf539 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:10.135200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b257c35-22b0-4120-ac05-0798a8ccf539 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:10.236462 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 519/1033] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:06:09 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:06:10.242088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2faf794f-29bb-493b-8d93-ae4db7aa22ca None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:10.287528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:10.355556 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2faf794f-29bb-493b-8d93-ae4db7aa22ca None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:10.381675 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7ed85e-0bd8-4913-8a66-95a3aa110cae None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:10.381962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7ed85e-0bd8-4913-8a66-95a3aa110cae None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:10.382235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7ed85e-0bd8-4913-8a66-95a3aa110cae None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:10.383592 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 520/1034] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:06:10.388561 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6d47f1-43fe-4462-a474-47b97495a41d None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:10.388851 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6d47f1-43fe-4462-a474-47b97495a41d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:10.389120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6d47f1-43fe-4462-a474-47b97495a41d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:10.391251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceaa9bd3-0e7a-4712-803e-b49a75bbaa28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:10.391946 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceaa9bd3-0e7a-4712-803e-b49a75bbaa28 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:10.392252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceaa9bd3-0e7a-4712-803e-b49a75bbaa28 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:10.392515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceaa9bd3-0e7a-4712-803e-b49a75bbaa28 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:10.407647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceaa9bd3-0e7a-4712-803e-b49a75bbaa28 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:10.409123 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceaa9bd3-0e7a-4712-803e-b49a75bbaa28 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:10.410259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-5f6d47f1-43fe-4462-a474-47b97495a41d None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:10.411659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5f6d47f1-43fe-4462-a474-47b97495a41d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:10.419380 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:10.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-ceaa9bd3-0e7a-4712-803e-b49a75bbaa28 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:10.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:10.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:10.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:10.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:10.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:10.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:10.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:10.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:10.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:10.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:10.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:10.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:06:10.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:06:10.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:06:10.426674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:10.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:06:10.433973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:06:10.433973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:10.433973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:10.433973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 515/1035] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:10.433973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceaa9bd3-0e7a-4712-803e-b49a75bbaa28 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:10.496127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 521/1036] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:10 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:06:10.533139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:10.539615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceaa9bd3-0e7a-4712-803e-b49a75bbaa28 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:10.573574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac8f07c-8a49-4e05-a280-77fff2c3c535 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:10.574169 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac8f07c-8a49-4e05-a280-77fff2c3c535 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:10.574444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac8f07c-8a49-4e05-a280-77fff2c3c535 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:10.574710 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac8f07c-8a49-4e05-a280-77fff2c3c535 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:10.575334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac8f07c-8a49-4e05-a280-77fff2c3c535 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:10.575630 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac8f07c-8a49-4e05-a280-77fff2c3c535 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:10.577049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac8f07c-8a49-4e05-a280-77fff2c3c535 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:10.588649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 516/1037] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:06:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:10.594741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6d47f1-43fe-4462-a474-47b97495a41d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:10.697743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6d47f1-43fe-4462-a474-47b97495a41d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:10.731290 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7674cd-6819-4474-bbdd-225b04119eca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:10.731949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7674cd-6819-4474-bbdd-225b04119eca None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:10.732275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7674cd-6819-4474-bbdd-225b04119eca None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:10.732551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7674cd-6819-4474-bbdd-225b04119eca None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:10.739795 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7674cd-6819-4474-bbdd-225b04119eca None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:10.741272 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7674cd-6819-4474-bbdd-225b04119eca None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:10.752497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-bb7674cd-6819-4474-bbdd-225b04119eca None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:06:10.752497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:10.752497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:10.752497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:10.752497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:10.752497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:10.752497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:10.752497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:10.752497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:10.752497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:10.752497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:10.752497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:10.752497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:06:10.752497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:06:10.752497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:10.753420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:10.759361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:06:10.759361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:06:10.759361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:06:10.759361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:06:10.759361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:06:10.759361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:06:10.759361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:06:10.759361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:10.759361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 522/1038] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:06:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:10.760031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac8f07c-8a49-4e05-a280-77fff2c3c535 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:10.761323 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7674cd-6819-4474-bbdd-225b04119eca None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:10.764151 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:10.861919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7674cd-6819-4474-bbdd-225b04119eca None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:10.862574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac8f07c-8a49-4e05-a280-77fff2c3c535 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:10.900547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c317d9-48e9-487c-ab39-f9e59f7ce0a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:10.901189 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b410ba-30c1-4fb8-8349-6af52523a86b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:10.901189 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b410ba-30c1-4fb8-8349-6af52523a86b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:10.901377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c317d9-48e9-487c-ab39-f9e59f7ce0a6 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:10.901468 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b410ba-30c1-4fb8-8349-6af52523a86b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:10.901551 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c317d9-48e9-487c-ab39-f9e59f7ce0a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:10.901834 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b410ba-30c1-4fb8-8349-6af52523a86b None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:10.901936 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c317d9-48e9-487c-ab39-f9e59f7ce0a6 None admin] PATH_INFO: `/v3/services/image` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:10.902257 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c317d9-48e9-487c-ab39-f9e59f7ce0a6 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:10.902363 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b410ba-30c1-4fb8-8349-6af52523a86b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:10.902639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b410ba-30c1-4fb8-8349-6af52523a86b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:10.903634 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c317d9-48e9-487c-ab39-f9e59f7ce0a6 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:10.904023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b410ba-30c1-4fb8-8349-6af52523a86b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-32c317d9-48e9-487c-ab39-f9e59f7ce0a6 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:10.913031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:10.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:10.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:10.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:10.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:10.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:10.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:10.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:06:10.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:06:10.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:06:10.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:06:10.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:10.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:10.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:10.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:10.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:10.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:10.914992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:10.914992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:10.914992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:10.914992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:10.914992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:10.914992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:10.914992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:10.914992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:10.914992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:10.914992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:10.914992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:06:10.914992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:06:10.914992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 21 01:06:10.914992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:10.914992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 517/1039] 10.222.0.47 () {56 vars in 1105 bytes} [Sun Aug 21 01:06:10 2022] GET /identity/v3/services/image => generated 86 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:10.924776 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 523/1040] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:06:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:10.924914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c317d9-48e9-487c-ab39-f9e59f7ce0a6 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:10.925553 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b410ba-30c1-4fb8-8349-6af52523a86b None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:10.944518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:10.991389 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:11.031730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c317d9-48e9-487c-ab39-f9e59f7ce0a6 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:11.034335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b410ba-30c1-4fb8-8349-6af52523a86b None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:11.066807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7f0163-a4cb-4e08-bee6-4a89aaa5453d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:11.067415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7f0163-a4cb-4e08-bee6-4a89aaa5453d None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:11.067683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7f0163-a4cb-4e08-bee6-4a89aaa5453d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:11.067955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7f0163-a4cb-4e08-bee6-4a89aaa5453d None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:11.068600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7f0163-a4cb-4e08-bee6-4a89aaa5453d None admin] RBAC: Adding query filter params (name=image) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:11.068894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7f0163-a4cb-4e08-bee6-4a89aaa5453d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:11.069435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6793bb-fb18-4089-a932-6fcd24c0fdbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:11.070032 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6793bb-fb18-4089-a932-6fcd24c0fdbe None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:11.070248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7f0163-a4cb-4e08-bee6-4a89aaa5453d None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:11.070359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6793bb-fb18-4089-a932-6fcd24c0fdbe None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:11.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6793bb-fb18-4089-a932-6fcd24c0fdbe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:11.071265 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6793bb-fb18-4089-a932-6fcd24c0fdbe None admin] RBAC: Adding query filter params (role.id=830cf269498649038ec8a27a3712fc60, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=fabdcdc25d1d4ef6bc66bd6cadecaaf8) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:11.071565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6793bb-fb18-4089-a932-6fcd24c0fdbe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:11.072976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6793bb-fb18-4089-a932-6fcd24c0fdbe None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:11.076213 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 518/1041] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:10 2022] GET /identity/v3/services?name=image => generated 122 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:11.081476 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7f0163-a4cb-4e08-bee6-4a89aaa5453d None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:11.083327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 524/1042] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:06:10 2022] GET /identity/v3/role_assignments?user.id=fabdcdc25d1d4ef6bc66bd6cadecaaf8&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=830cf269498649038ec8a27a3712fc60 => generated 259 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:11.175033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:11.195578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7f0163-a4cb-4e08-bee6-4a89aaa5453d None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:11.235003 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06351a7c-4235-4a0b-8f88-54a7bcf7107a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:11.235342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06351a7c-4235-4a0b-8f88-54a7bcf7107a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:11.235601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06351a7c-4235-4a0b-8f88-54a7bcf7107a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:11.236884 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06351a7c-4235-4a0b-8f88-54a7bcf7107a None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:11.236884 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06351a7c-4235-4a0b-8f88-54a7bcf7107a None admin] RBAC: Adding query filter params (type=image) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:11.236884 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06351a7c-4235-4a0b-8f88-54a7bcf7107a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:11.238157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06351a7c-4235-4a0b-8f88-54a7bcf7107a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:11.243143 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 519/1043] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:11 2022] GET /identity/v3/services?type=image => generated 350 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:11.249374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6793bb-fb18-4089-a932-6fcd24c0fdbe None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:11.360435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6793bb-fb18-4089-a932-6fcd24c0fdbe None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:11.385716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:11.394936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec62d8d-337e-40d1-b79b-453a8c15fb4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:11.395544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec62d8d-337e-40d1-b79b-453a8c15fb4e None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:11.395816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec62d8d-337e-40d1-b79b-453a8c15fb4e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:11.396091 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec62d8d-337e-40d1-b79b-453a8c15fb4e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:11.396805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec62d8d-337e-40d1-b79b-453a8c15fb4e None admin] RBAC: Adding query filter params (interface=public, service_id=8c37d0250a1d490c906cbe905e137b65, region_id=RegionOne) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:11.397106 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec62d8d-337e-40d1-b79b-453a8c15fb4e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:11.398458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec62d8d-337e-40d1-b79b-453a8c15fb4e None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:11.408162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 525/1044] 10.222.0.47 () {56 vars in 1251 bytes} [Sun Aug 21 01:06:11 2022] GET /identity/v3/endpoints?service_id=8c37d0250a1d490c906cbe905e137b65&interface=public®ion_id=RegionOne => generated 194 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:11.410282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06351a7c-4235-4a0b-8f88-54a7bcf7107a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:11.511830 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06351a7c-4235-4a0b-8f88-54a7bcf7107a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:11.546867 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38608b47-eb2b-4118-9396-57174c8e472c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:11.547470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38608b47-eb2b-4118-9396-57174c8e472c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:11.547745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38608b47-eb2b-4118-9396-57174c8e472c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:11.548014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38608b47-eb2b-4118-9396-57174c8e472c None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:11.548653 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38608b47-eb2b-4118-9396-57174c8e472c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:11.549998 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38608b47-eb2b-4118-9396-57174c8e472c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:11.564147 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 520/1045] 10.222.0.47 () {56 vars in 1087 bytes} [Sun Aug 21 01:06:11 2022] GET /identity/v3/services => generated 1957 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:06:11.720301 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9195937-6527-4540-b97d-c65a4ab885b1 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:11.720625 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9195937-6527-4540-b97d-c65a4ab885b1 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:11.720848 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9195937-6527-4540-b97d-c65a4ab885b1 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:11.722183 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 526/1046] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:11.728151 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660fbd00-259a-4af5-bb98-0e454910c5c3 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:11.728443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660fbd00-259a-4af5-bb98-0e454910c5c3 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:11.728721 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660fbd00-259a-4af5-bb98-0e454910c5c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:11.747464 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-660fbd00-259a-4af5-bb98-0e454910c5c3 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:11.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-660fbd00-259a-4af5-bb98-0e454910c5c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:11.753649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:11.820272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 521/1047] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:11 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:12.088321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9195937-6527-4540-b97d-c65a4ab885b1 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:12.196761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9195937-6527-4540-b97d-c65a4ab885b1 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:12.232892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcbc6d9-2c5c-4d35-9ac2-8212df6c45c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:12.234063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcbc6d9-2c5c-4d35-9ac2-8212df6c45c7 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:12.234339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcbc6d9-2c5c-4d35-9ac2-8212df6c45c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:12.234609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcbc6d9-2c5c-4d35-9ac2-8212df6c45c7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:12.244770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fcbc6d9-2c5c-4d35-9ac2-8212df6c45c7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:06:12.249483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-9fcbc6d9-2c5c-4d35-9ac2-8212df6c45c7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:06:12.249483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:12.249483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:12.249483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:12.249483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:12.249483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:12.249483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:12.249483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:12.249483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:12.249483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:12.249483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:12.249483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:12.249483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:06:12.249483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:06:12.249483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:06:12.249483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:06:12.250476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:06:12.250476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:06:12.250476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:06:12.250476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:06:12.250476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:12.250476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:12.250476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:12.250476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:12.250476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:12.250476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:12.250476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:12.250476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:12.250476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:12.250476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:12.250476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:12.250476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:12.252950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:12.252950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:12.252950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:12.252950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:12.252950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:06:12.252950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:06:12.252950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:06:12.252950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:06:12.252950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:06:12.252950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:06:12.252950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:06:12.252950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:12.252950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 527/1048] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:06:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:12.256996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660fbd00-259a-4af5-bb98-0e454910c5c3 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:12.366388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660fbd00-259a-4af5-bb98-0e454910c5c3 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:12.401801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bdfb89-6ade-4c78-93b7-cec33eec1556 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:12.402413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bdfb89-6ade-4c78-93b7-cec33eec1556 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:12.402740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bdfb89-6ade-4c78-93b7-cec33eec1556 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:12.403011 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bdfb89-6ade-4c78-93b7-cec33eec1556 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:12.403665 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bdfb89-6ade-4c78-93b7-cec33eec1556 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:12.403957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bdfb89-6ade-4c78-93b7-cec33eec1556 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:12.405963 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bdfb89-6ade-4c78-93b7-cec33eec1556 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:12.419374 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 522/1049] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:06:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:12.420821 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcbc6d9-2c5c-4d35-9ac2-8212df6c45c7 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:12.473705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:12.523261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcbc6d9-2c5c-4d35-9ac2-8212df6c45c7 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:12.557483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798ec449-a93a-4719-8aeb-4d823bbc8ad9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:12.558116 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798ec449-a93a-4719-8aeb-4d823bbc8ad9 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:12.558395 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798ec449-a93a-4719-8aeb-4d823bbc8ad9 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:12.558663 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798ec449-a93a-4719-8aeb-4d823bbc8ad9 None admin] PATH_INFO: `/v3/users/58d95df9c2014d63aefb9311e852e828` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:12.573192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798ec449-a93a-4719-8aeb-4d823bbc8ad9 None admin] RBAC: Authorizing `identity:get_user(user_id=58d95df9c2014d63aefb9311e852e828)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:12.574622 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798ec449-a93a-4719-8aeb-4d823bbc8ad9 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:12.581671 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 528/1050] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:06:12 2022] GET /identity/v3/users/58d95df9c2014d63aefb9311e852e828 => generated 285 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:12.583186 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bdfb89-6ade-4c78-93b7-cec33eec1556 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:12.586164 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:12.688396 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bdfb89-6ade-4c78-93b7-cec33eec1556 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:12.722989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a31ec4-2cf7-4065-8003-266aec65888d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:12.723605 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a31ec4-2cf7-4065-8003-266aec65888d None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:12.723953 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a31ec4-2cf7-4065-8003-266aec65888d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:12.724193 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a31ec4-2cf7-4065-8003-266aec65888d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:12.791078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a31ec4-2cf7-4065-8003-266aec65888d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:12.792547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a31ec4-2cf7-4065-8003-266aec65888d None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:12.794358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:12.897313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 523/1051] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:06:12 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 21 01:06:12.899204 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798ec449-a93a-4719-8aeb-4d823bbc8ad9 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:12.993000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b84171a-6066-474e-bc9c-1246ba26b68d None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:12.993288 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b84171a-6066-474e-bc9c-1246ba26b68d None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:12.993722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b84171a-6066-474e-bc9c-1246ba26b68d None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:12.995312 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 524/1052] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:06:13.000985 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798ec449-a93a-4719-8aeb-4d823bbc8ad9 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:13.004300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811e3298-28dc-419c-b84e-e3489cd89e96 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:13.004796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811e3298-28dc-419c-b84e-e3489cd89e96 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:13.005290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811e3298-28dc-419c-b84e-e3489cd89e96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:13.010647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:13.029248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-811e3298-28dc-419c-b84e-e3489cd89e96 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:13.031248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-811e3298-28dc-419c-b84e-e3489cd89e96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:13.035799 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4e1fe0-9a20-4162-8ef7-bf136eda67fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:13.036619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4e1fe0-9a20-4162-8ef7-bf136eda67fd None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:13.037188 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4e1fe0-9a20-4162-8ef7-bf136eda67fd None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:13.037460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4e1fe0-9a20-4162-8ef7-bf136eda67fd None admin] PATH_INFO: `/v3/projects/ef43bd38583d4e5794474db52b2f8958` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:13.038933 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4e1fe0-9a20-4162-8ef7-bf136eda67fd None admin] RBAC: Authorizing `identity:get_project(project_id=ef43bd38583d4e5794474db52b2f8958)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:13.040047 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4e1fe0-9a20-4162-8ef7-bf136eda67fd None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:13.047269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 529/1053] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:06:12 2022] GET /identity/v3/projects/ef43bd38583d4e5794474db52b2f8958 => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:13.049064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4e1fe0-9a20-4162-8ef7-bf136eda67fd None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:13.111463 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 525/1054] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:13 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:06:13.153504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4e1fe0-9a20-4162-8ef7-bf136eda67fd None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:13.188840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fef98d-d5e0-49b8-9605-420306a0f8b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:13.189703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fef98d-d5e0-49b8-9605-420306a0f8b5 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:13.189980 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fef98d-d5e0-49b8-9605-420306a0f8b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:13.190248 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fef98d-d5e0-49b8-9605-420306a0f8b5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:13.190962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fef98d-d5e0-49b8-9605-420306a0f8b5 None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=ef43bd38583d4e5794474db52b2f8958, user.id=58d95df9c2014d63aefb9311e852e828) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:13.191266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fef98d-d5e0-49b8-9605-420306a0f8b5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:13.192699 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fef98d-d5e0-49b8-9605-420306a0f8b5 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:13.198262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 530/1055] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:06:13 2022] GET /identity/v3/role_assignments?user.id=58d95df9c2014d63aefb9311e852e828&scope.project.id=ef43bd38583d4e5794474db52b2f8958&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 259 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:13.385250 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811e3298-28dc-419c-b84e-e3489cd89e96 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:13.417076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:13.470348 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ce8056-44ff-49d3-b04a-5a0852121355 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:13.470707 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ce8056-44ff-49d3-b04a-5a0852121355 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:13.470909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ce8056-44ff-49d3-b04a-5a0852121355 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:13.472273 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 531/1056] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:06:13.477169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c10b27-69ad-4526-80a6-76f10d3e9e32 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:13.477452 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c10b27-69ad-4526-80a6-76f10d3e9e32 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:13.477723 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c10b27-69ad-4526-80a6-76f10d3e9e32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:13.491116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811e3298-28dc-419c-b84e-e3489cd89e96 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:13.495630 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-46c10b27-69ad-4526-80a6-76f10d3e9e32 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:13.496837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-46c10b27-69ad-4526-80a6-76f10d3e9e32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:13.527934 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c64e828-731d-41f6-86e9-caac25c3a086 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:13.528604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c64e828-731d-41f6-86e9-caac25c3a086 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:13.528876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c64e828-731d-41f6-86e9-caac25c3a086 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:13.529144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c64e828-731d-41f6-86e9-caac25c3a086 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:13.532579 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:13.539919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c64e828-731d-41f6-86e9-caac25c3a086 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:06:13.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-3c64e828-731d-41f6-86e9-caac25c3a086 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 21 01:06:13.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:13.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:13.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:13.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:13.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:13.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:13.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:13.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:13.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:13.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:13.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:13.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:06:13.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:06:13.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:06:13.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:06:13.547928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:06:13.547928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:06:13.547928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:06:13.547928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:06:13.547928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:13.547928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:13.547928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:13.547928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:13.547928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:13.547928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:13.547928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:13.547928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:13.547928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:13.547928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:13.547928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:13.547928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:13.548832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:13.548832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:13.548832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:13.548832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:13.548832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:06:13.548832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:06:13.548832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:06:13.548832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:06:13.548832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:06:13.548832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:06:13.548832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 21 01:06:13.548832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:13.548832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 526/1057] 10.222.0.47 () {56 vars in 1120 bytes} [Sun Aug 21 01:06:13 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:06:13.551783 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c64e828-731d-41f6-86e9-caac25c3a086 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:13.572197 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 532/1058] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:13 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:06:13.646555 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:13.655894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c64e828-731d-41f6-86e9-caac25c3a086 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:13.690832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf33591c-ba66-415e-8ec5-08b059db6442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:13.691443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf33591c-ba66-415e-8ec5-08b059db6442 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:13.691723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf33591c-ba66-415e-8ec5-08b059db6442 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:13.691996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf33591c-ba66-415e-8ec5-08b059db6442 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:13.692688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf33591c-ba66-415e-8ec5-08b059db6442 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:13.693008 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf33591c-ba66-415e-8ec5-08b059db6442 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:13.694401 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf33591c-ba66-415e-8ec5-08b059db6442 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:13.707598 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 527/1059] 10.222.0.47 () {56 vars in 1115 bytes} [Sun Aug 21 01:06:13 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:13.708829 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c10b27-69ad-4526-80a6-76f10d3e9e32 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:13.817899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c10b27-69ad-4526-80a6-76f10d3e9e32 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:13.846213 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf33591c-ba66-415e-8ec5-08b059db6442 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:13.851833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1597fd05-b47b-41c3-8b49-3f468eccfc7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:13.852493 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1597fd05-b47b-41c3-8b49-3f468eccfc7b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:13.852778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1597fd05-b47b-41c3-8b49-3f468eccfc7b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:13.853049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1597fd05-b47b-41c3-8b49-3f468eccfc7b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:13.863723 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1597fd05-b47b-41c3-8b49-3f468eccfc7b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:13.865504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1597fd05-b47b-41c3-8b49-3f468eccfc7b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:13.880154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-1597fd05-b47b-41c3-8b49-3f468eccfc7b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:13.880154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:13.880154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:13.880154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:13.880154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:13.880154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:13.880154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:13.880154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:13.880154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:13.880154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:13.880154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:13.880154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:13.880154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:06:13.880154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:06:13.880154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:06:13.880154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:13.881144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:06:13.882435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:06:13.882435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:13.882435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:13.882435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 533/1060] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:13.886812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1597fd05-b47b-41c3-8b49-3f468eccfc7b None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:13.912782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:13.962214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf33591c-ba66-415e-8ec5-08b059db6442 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:13.996239 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1597fd05-b47b-41c3-8b49-3f468eccfc7b None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:13.996664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd6407a-3c18-4eea-bc6c-64512755f612 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:13.997279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd6407a-3c18-4eea-bc6c-64512755f612 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:13.997549 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd6407a-3c18-4eea-bc6c-64512755f612 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:13.997820 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd6407a-3c18-4eea-bc6c-64512755f612 None admin] PATH_INFO: `/v3/services/image` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:13.998395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd6407a-3c18-4eea-bc6c-64512755f612 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:14.000388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd6407a-3c18-4eea-bc6c-64512755f612 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-4dd6407a-3c18-4eea-bc6c-64512755f612 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:14.009260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:14.012300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:14.012300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:14.012300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:14.012300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:14.012300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:14.012300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:14.012300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:06:14.012300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:06:14.012300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:06:14.012300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:06:14.012300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:14.012300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:14.012300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:14.012300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:14.012300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:14.012300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:14.013155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:14.013155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:14.013155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:14.013155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:14.013155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:14.013155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:14.013155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:14.013155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:14.013155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:14.013155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:14.013155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:06:14.013155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:06:14.013155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 21 01:06:14.013155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:14.013155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 528/1061] 10.222.0.47 () {56 vars in 1105 bytes} [Sun Aug 21 01:06:13 2022] GET /identity/v3/services/image => generated 86 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:06:14.016242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd6407a-3c18-4eea-bc6c-64512755f612 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:14.036700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:14.037836 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a40b979-acec-413e-b0c3-e48ffffb0a32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:14.038446 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a40b979-acec-413e-b0c3-e48ffffb0a32 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:14.038716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a40b979-acec-413e-b0c3-e48ffffb0a32 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:14.038983 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a40b979-acec-413e-b0c3-e48ffffb0a32 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:14.039618 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a40b979-acec-413e-b0c3-e48ffffb0a32 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:14.039912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a40b979-acec-413e-b0c3-e48ffffb0a32 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:14.041387 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a40b979-acec-413e-b0c3-e48ffffb0a32 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:14.054066 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 534/1062] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:06:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:14.059307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a40b979-acec-413e-b0c3-e48ffffb0a32 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:14.130950 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd6407a-3c18-4eea-bc6c-64512755f612 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:14.149220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:14.167874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80edc5b9-2ce0-4fdc-b1a7-6507468bde41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:14.168560 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80edc5b9-2ce0-4fdc-b1a7-6507468bde41 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:14.186737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80edc5b9-2ce0-4fdc-b1a7-6507468bde41 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:14.186865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a40b979-acec-413e-b0c3-e48ffffb0a32 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:14.187283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80edc5b9-2ce0-4fdc-b1a7-6507468bde41 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:14.187283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80edc5b9-2ce0-4fdc-b1a7-6507468bde41 None admin] RBAC: Adding query filter params (name=image) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:14.187283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80edc5b9-2ce0-4fdc-b1a7-6507468bde41 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:14.187283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80edc5b9-2ce0-4fdc-b1a7-6507468bde41 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:14.187283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 529/1063] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:14 2022] GET /identity/v3/services?name=image => generated 122 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:14.187283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80edc5b9-2ce0-4fdc-b1a7-6507468bde41 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:14.206934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d39196d-a88d-4347-940c-96fcadab5a54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:14.207985 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d39196d-a88d-4347-940c-96fcadab5a54 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:14.208455 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d39196d-a88d-4347-940c-96fcadab5a54 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:14.209103 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d39196d-a88d-4347-940c-96fcadab5a54 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:14.216231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d39196d-a88d-4347-940c-96fcadab5a54 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:14.218195 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d39196d-a88d-4347-940c-96fcadab5a54 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:14.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-2d39196d-a88d-4347-940c-96fcadab5a54 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:06:14.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:14.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:14.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:14.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:14.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:14.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:14.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:14.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:14.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:14.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:14.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:14.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:06:14.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:06:14.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:06:14.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:14.230696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:14.231824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:14.231824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:14.231824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:06:14.231824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:06:14.231824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:06:14.231824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:06:14.231824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:06:14.231824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:06:14.231824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:06:14.231824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:14.231824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 535/1064] 10.222.0.47 () {56 vars in 1093 bytes} [Sun Aug 21 01:06:14 2022] GET /identity/v3/users/nova => generated 82 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:06:14.236897 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d39196d-a88d-4347-940c-96fcadab5a54 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:14.280403 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:14.299794 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80edc5b9-2ce0-4fdc-b1a7-6507468bde41 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:14.337546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400117df-70b4-4fa5-b398-8e06a540df0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:14.338151 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400117df-70b4-4fa5-b398-8e06a540df0f None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:14.338421 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400117df-70b4-4fa5-b398-8e06a540df0f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:14.338687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400117df-70b4-4fa5-b398-8e06a540df0f None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:14.339318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400117df-70b4-4fa5-b398-8e06a540df0f None admin] RBAC: Adding query filter params (type=image) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:14.339614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400117df-70b4-4fa5-b398-8e06a540df0f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:14.340985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400117df-70b4-4fa5-b398-8e06a540df0f None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:14.346713 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 530/1065] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:14 2022] GET /identity/v3/services?type=image => generated 350 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:14.352875 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400117df-70b4-4fa5-b398-8e06a540df0f None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:14.363860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d39196d-a88d-4347-940c-96fcadab5a54 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:14.400542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9468b179-6659-44d2-8fc5-66412ec6e76b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:14.401366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9468b179-6659-44d2-8fc5-66412ec6e76b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:14.401641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9468b179-6659-44d2-8fc5-66412ec6e76b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:14.401911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9468b179-6659-44d2-8fc5-66412ec6e76b None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:14.402651 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9468b179-6659-44d2-8fc5-66412ec6e76b None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:14.402953 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9468b179-6659-44d2-8fc5-66412ec6e76b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:14.404401 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9468b179-6659-44d2-8fc5-66412ec6e76b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:14.419314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:14.427672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 536/1066] 10.222.0.47 () {56 vars in 1133 bytes} [Sun Aug 21 01:06:14 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:14.435261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9468b179-6659-44d2-8fc5-66412ec6e76b None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:14.463338 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400117df-70b4-4fa5-b398-8e06a540df0f None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:14.499063 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c69d5ae-27b9-404f-945a-9f1b60f61c68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:14.499669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c69d5ae-27b9-404f-945a-9f1b60f61c68 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:14.499943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c69d5ae-27b9-404f-945a-9f1b60f61c68 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:14.500235 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c69d5ae-27b9-404f-945a-9f1b60f61c68 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:14.500890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c69d5ae-27b9-404f-945a-9f1b60f61c68 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:14.502242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c69d5ae-27b9-404f-945a-9f1b60f61c68 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:14.530942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9468b179-6659-44d2-8fc5-66412ec6e76b None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:14.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674a4856-1318-4b7b-9bc3-9c8e01128c6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:14.566272 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674a4856-1318-4b7b-9bc3-9c8e01128c6e None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:14.566550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674a4856-1318-4b7b-9bc3-9c8e01128c6e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:14.566815 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674a4856-1318-4b7b-9bc3-9c8e01128c6e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:14.576619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674a4856-1318-4b7b-9bc3-9c8e01128c6e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:14.578036 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674a4856-1318-4b7b-9bc3-9c8e01128c6e None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:14.588923 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 531/1067] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:06:14 2022] POST /identity/v3/endpoints => generated 327 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:06:14.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-674a4856-1318-4b7b-9bc3-9c8e01128c6e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:14.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:14.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:14.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:14.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:14.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:14.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:14.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:14.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:14.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:14.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:14.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:14.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:06:14.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:06:14.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:06:14.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:14.598813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:06:14.604945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:06:14.604945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:14.604945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:14.604945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 537/1068] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:14.606827 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c69d5ae-27b9-404f-945a-9f1b60f61c68 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:14.610456 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:14.724846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c69d5ae-27b9-404f-945a-9f1b60f61c68 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:14.763152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99c5d82-2269-4d55-8a6d-c6b5aa98b65d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:14.763761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99c5d82-2269-4d55-8a6d-c6b5aa98b65d None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:14.764215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99c5d82-2269-4d55-8a6d-c6b5aa98b65d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:14.764433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99c5d82-2269-4d55-8a6d-c6b5aa98b65d None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:14.765033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99c5d82-2269-4d55-8a6d-c6b5aa98b65d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:14.765339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99c5d82-2269-4d55-8a6d-c6b5aa98b65d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:14.766700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99c5d82-2269-4d55-8a6d-c6b5aa98b65d None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:14.778853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 532/1069] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:06:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:14.784724 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674a4856-1318-4b7b-9bc3-9c8e01128c6e None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:14.843892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:14.892548 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674a4856-1318-4b7b-9bc3-9c8e01128c6e None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:14.926917 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2ff611-35f2-4c9e-a695-8f3cdbe7fe47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:14.927544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ff611-35f2-4c9e-a695-8f3cdbe7fe47 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:14.927821 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ff611-35f2-4c9e-a695-8f3cdbe7fe47 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:14.928088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ff611-35f2-4c9e-a695-8f3cdbe7fe47 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:14.935025 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2ff611-35f2-4c9e-a695-8f3cdbe7fe47 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:14.936514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2ff611-35f2-4c9e-a695-8f3cdbe7fe47 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:14.947911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-4e2ff611-35f2-4c9e-a695-8f3cdbe7fe47 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:06:14.947911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:14.947911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:14.947911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:14.947911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:14.947911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:14.947911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:14.947911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:14.947911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:14.947911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:14.947911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:14.947911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:14.947911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:06:14.947911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:06:14.947911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:14.948871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:14.954956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:06:14.954956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:06:14.954956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:06:14.954956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:06:14.954956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:06:14.954956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:06:14.954956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:06:14.954956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:14.954956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 538/1070] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:06:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:06:14.957078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99c5d82-2269-4d55-8a6d-c6b5aa98b65d None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:15.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:15.060482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99c5d82-2269-4d55-8a6d-c6b5aa98b65d None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:15.096906 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8469b13e-bc0a-4e3d-88fe-96960fd164cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:15.097506 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8469b13e-bc0a-4e3d-88fe-96960fd164cf None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:15.097782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8469b13e-bc0a-4e3d-88fe-96960fd164cf None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:15.098041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8469b13e-bc0a-4e3d-88fe-96960fd164cf None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:15.098717 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8469b13e-bc0a-4e3d-88fe-96960fd164cf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:15.099041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8469b13e-bc0a-4e3d-88fe-96960fd164cf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:15.100478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8469b13e-bc0a-4e3d-88fe-96960fd164cf None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:15.106286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37ae355-0e03-4fc8-9eae-2139c528bbf5 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:15.106569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37ae355-0e03-4fc8-9eae-2139c528bbf5 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:15.106844 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37ae355-0e03-4fc8-9eae-2139c528bbf5 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:15.109053 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 539/1071] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:06:15.117535 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 533/1072] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:06:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:15.117701 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8c5787-7594-403a-828e-10f1b4fd0b31 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:15.117701 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8c5787-7594-403a-828e-10f1b4fd0b31 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:15.117701 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8c5787-7594-403a-828e-10f1b4fd0b31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:15.119380 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8469b13e-bc0a-4e3d-88fe-96960fd164cf None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:15.132659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-ab8c5787-7594-403a-828e-10f1b4fd0b31 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:15.133850 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ab8c5787-7594-403a-828e-10f1b4fd0b31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:15.205580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 540/1073] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:15 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:06:15.221016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8469b13e-bc0a-4e3d-88fe-96960fd164cf None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:15.238195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:15.255676 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d003246-42f4-46e4-acbf-66cab58a7ed8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:15.256497 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d003246-42f4-46e4-acbf-66cab58a7ed8 None admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:15.256878 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d003246-42f4-46e4-acbf-66cab58a7ed8 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:15.257237 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d003246-42f4-46e4-acbf-66cab58a7ed8 None admin] PATH_INFO: `/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/fabdcdc25d1d4ef6bc66bd6cadecaaf8/roles/830cf269498649038ec8a27a3712fc60` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:15.259458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d003246-42f4-46e4-acbf-66cab58a7ed8 None admin] RBAC: Authorizing `identity:create_grant(project_id=75a3bf3f3a4244d18fde47d492657db4, user_id=fabdcdc25d1d4ef6bc66bd6cadecaaf8, role_id=830cf269498649038ec8a27a3712fc60)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:15.261047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d003246-42f4-46e4-acbf-66cab58a7ed8 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:15.319016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 534/1074] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:06:15 2022] PUT /identity/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/fabdcdc25d1d4ef6bc66bd6cadecaaf8/roles/830cf269498649038ec8a27a3712fc60 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:06:15.477746 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8c5787-7594-403a-828e-10f1b4fd0b31 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:15.524085 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:15.582138 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8c5787-7594-403a-828e-10f1b4fd0b31 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:15.619377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6983b0e-1eda-43ef-8008-7818f9f2f49d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:15.620004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6983b0e-1eda-43ef-8008-7818f9f2f49d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:15.620302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6983b0e-1eda-43ef-8008-7818f9f2f49d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:15.620575 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6983b0e-1eda-43ef-8008-7818f9f2f49d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:15.631382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6983b0e-1eda-43ef-8008-7818f9f2f49d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:06:15.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-d6983b0e-1eda-43ef-8008-7818f9f2f49d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:06:15.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:15.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:15.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:15.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:15.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:15.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:15.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:15.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:15.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:15.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:15.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:15.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:06:15.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:06:15.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:06:15.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:06:15.637101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:06:15.637101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:06:15.637101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:06:15.637101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:06:15.637101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:15.637101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:15.637101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:15.637101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:15.637101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:15.637101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:15.637101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:15.637101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:15.637101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:15.637101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:15.637101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:15.637101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:15.637948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:15.637948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:15.637948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:15.637948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:15.637948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:06:15.637948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:06:15.637948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:06:15.637948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:06:15.637948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:06:15.637948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:06:15.637948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:06:15.637948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:15.637948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 541/1075] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:06:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:15.644600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d003246-42f4-46e4-acbf-66cab58a7ed8 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:15.749783 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d003246-42f4-46e4-acbf-66cab58a7ed8 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:15.784949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c85d13b-edb6-4a19-951a-abe99efc6f42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:15.785556 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c85d13b-edb6-4a19-951a-abe99efc6f42 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:15.785826 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c85d13b-edb6-4a19-951a-abe99efc6f42 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:15.786088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c85d13b-edb6-4a19-951a-abe99efc6f42 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:15.786739 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c85d13b-edb6-4a19-951a-abe99efc6f42 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:15.787031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c85d13b-edb6-4a19-951a-abe99efc6f42 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:15.788401 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c85d13b-edb6-4a19-951a-abe99efc6f42 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:15.802303 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 535/1076] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:06:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:15.803643 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6983b0e-1eda-43ef-8008-7818f9f2f49d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:15.906589 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6983b0e-1eda-43ef-8008-7818f9f2f49d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:15.908049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:15.942434 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f85b6c-510e-486c-8cba-f4059002008d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:15.943068 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f85b6c-510e-486c-8cba-f4059002008d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:15.943345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f85b6c-510e-486c-8cba-f4059002008d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:15.943617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f85b6c-510e-486c-8cba-f4059002008d None admin] PATH_INFO: `/v3/users/58d95df9c2014d63aefb9311e852e828` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:15.944752 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f85b6c-510e-486c-8cba-f4059002008d None admin] RBAC: Authorizing `identity:get_user(user_id=58d95df9c2014d63aefb9311e852e828)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:15.946208 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f85b6c-510e-486c-8cba-f4059002008d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:15.952490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 542/1077] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:06:15 2022] GET /identity/v3/users/58d95df9c2014d63aefb9311e852e828 => generated 285 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:15.953666 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c85d13b-edb6-4a19-951a-abe99efc6f42 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:15.996768 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:16.056917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c85d13b-edb6-4a19-951a-abe99efc6f42 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:16.093930 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be68184-0561-4e40-be6f-3063136615d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:16.094555 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be68184-0561-4e40-be6f-3063136615d3 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:16.094827 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be68184-0561-4e40-be6f-3063136615d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:16.095092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be68184-0561-4e40-be6f-3063136615d3 None admin] PATH_INFO: `/v3/projects/ef43bd38583d4e5794474db52b2f8958` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:16.096234 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be68184-0561-4e40-be6f-3063136615d3 None admin] RBAC: Authorizing `identity:get_project(project_id=ef43bd38583d4e5794474db52b2f8958)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:16.097637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be68184-0561-4e40-be6f-3063136615d3 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:16.104144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 536/1078] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:06:15 2022] GET /identity/v3/projects/ef43bd38583d4e5794474db52b2f8958 => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:16.105291 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f85b6c-510e-486c-8cba-f4059002008d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:16.207865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f85b6c-510e-486c-8cba-f4059002008d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:16.244181 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337309e5-df72-4f89-b702-03635d6aecd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:16.244831 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337309e5-df72-4f89-b702-03635d6aecd8 None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:16.245103 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337309e5-df72-4f89-b702-03635d6aecd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:16.245370 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337309e5-df72-4f89-b702-03635d6aecd8 None admin] PATH_INFO: `/v3/projects/ef43bd38583d4e5794474db52b2f8958/users/58d95df9c2014d63aefb9311e852e828/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:16.247219 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337309e5-df72-4f89-b702-03635d6aecd8 None admin] RBAC: Authorizing `identity:create_grant(project_id=ef43bd38583d4e5794474db52b2f8958, user_id=58d95df9c2014d63aefb9311e852e828, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:16.248733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337309e5-df72-4f89-b702-03635d6aecd8 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:16.319669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 543/1079] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:06:16 2022] PUT /identity/v3/projects/ef43bd38583d4e5794474db52b2f8958/users/58d95df9c2014d63aefb9311e852e828/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:06:16.525875 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a596781f-d388-43e1-b7db-132b83ec41e1 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:16.526191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a596781f-d388-43e1-b7db-132b83ec41e1 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:16.526361 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a596781f-d388-43e1-b7db-132b83ec41e1 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:16.527825 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 537/1080] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:16.532898 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b1c561-ad96-42f6-8789-ae7de7ec4c6a None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:16.533184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b1c561-ad96-42f6-8789-ae7de7ec4c6a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:16.533458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b1c561-ad96-42f6-8789-ae7de7ec4c6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:16.551194 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-c0b1c561-ad96-42f6-8789-ae7de7ec4c6a None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:16.552402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-c0b1c561-ad96-42f6-8789-ae7de7ec4c6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:16.578061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:16.621784 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 544/1081] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:16.891043 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a596781f-d388-43e1-b7db-132b83ec41e1 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:16.993516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a596781f-d388-43e1-b7db-132b83ec41e1 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:17.002909 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:17.027981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3707726-b064-468b-bac7-1ac335f83449 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:17.028621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3707726-b064-468b-bac7-1ac335f83449 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:17.028909 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3707726-b064-468b-bac7-1ac335f83449 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:17.029178 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3707726-b064-468b-bac7-1ac335f83449 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:17.098985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3707726-b064-468b-bac7-1ac335f83449 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:17.100459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3707726-b064-468b-bac7-1ac335f83449 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:17.204135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 538/1082] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:06:16 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:06:17.208384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b1c561-ad96-42f6-8789-ae7de7ec4c6a None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:17.244517 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f105e775-ecf1-4dc6-8058-b017faf4819e None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:17.244884 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f105e775-ecf1-4dc6-8058-b017faf4819e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:17.245083 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f105e775-ecf1-4dc6-8058-b017faf4819e None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:17.246572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 539/1083] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:17.251690 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa52cf96-ae63-439f-8dc0-e9161ad04012 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:17.251973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa52cf96-ae63-439f-8dc0-e9161ad04012 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:17.252291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa52cf96-ae63-439f-8dc0-e9161ad04012 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:17.270852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-aa52cf96-ae63-439f-8dc0-e9161ad04012 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:17.272052 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-aa52cf96-ae63-439f-8dc0-e9161ad04012 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:17.274383 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:17.318448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b1c561-ad96-42f6-8789-ae7de7ec4c6a None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:17.353198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 540/1084] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:06:17.362881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af34df46-2787-4e4a-be14-5239e27cdc24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:17.363752 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af34df46-2787-4e4a-be14-5239e27cdc24 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:17.364263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af34df46-2787-4e4a-be14-5239e27cdc24 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:17.364711 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af34df46-2787-4e4a-be14-5239e27cdc24 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:17.375695 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af34df46-2787-4e4a-be14-5239e27cdc24 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:17.377431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af34df46-2787-4e4a-be14-5239e27cdc24 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:17.393580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-af34df46-2787-4e4a-be14-5239e27cdc24 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:17.393580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:17.393580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:17.393580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:17.393580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:17.393580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:17.393580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:17.393580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:17.393580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:17.393580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:17.393580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:17.393580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:17.393580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:06:17.393580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:06:17.393580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:06:17.393580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:17.394616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:06:17.395536 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:06:17.395536 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:17.395536 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:17.397480 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 545/1085] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:17.403123 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa52cf96-ae63-439f-8dc0-e9161ad04012 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:17.507950 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa52cf96-ae63-439f-8dc0-e9161ad04012 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:17.542799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98315bbe-db2d-4c85-9ba4-eaeaad848edf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:17.543593 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98315bbe-db2d-4c85-9ba4-eaeaad848edf None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:17.544021 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98315bbe-db2d-4c85-9ba4-eaeaad848edf None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:17.544481 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98315bbe-db2d-4c85-9ba4-eaeaad848edf None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:17.545290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98315bbe-db2d-4c85-9ba4-eaeaad848edf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:17.545755 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98315bbe-db2d-4c85-9ba4-eaeaad848edf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:17.547247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98315bbe-db2d-4c85-9ba4-eaeaad848edf None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:17.553821 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:17.559630 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 541/1086] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:06:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:17.626337 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af34df46-2787-4e4a-be14-5239e27cdc24 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:17.642923 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:17.737100 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af34df46-2787-4e4a-be14-5239e27cdc24 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:17.772762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28552780-539e-4d7b-b93b-0fdc5fab69a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:17.773325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28552780-539e-4d7b-b93b-0fdc5fab69a8 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:17.773607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28552780-539e-4d7b-b93b-0fdc5fab69a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:17.773881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28552780-539e-4d7b-b93b-0fdc5fab69a8 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:17.784701 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28552780-539e-4d7b-b93b-0fdc5fab69a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:06:17.789599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-28552780-539e-4d7b-b93b-0fdc5fab69a8 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 21 01:06:17.789599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:17.789599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:17.789599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:17.789599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:17.789599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:17.789599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:17.789599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:17.789599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:17.789599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:17.789599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:17.789599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:17.789599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:06:17.789599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:06:17.789599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:06:17.789599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:06:17.790547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:06:17.790547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:06:17.790547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:06:17.790547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:06:17.790547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:17.790547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:17.790547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:17.790547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:17.790547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:17.790547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:17.790547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:17.790547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:17.790547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:17.790547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:17.790547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:17.790547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:17.791587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:17.791587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:17.791587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:17.791587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:17.791587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:06:17.791587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:06:17.791587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:06:17.791587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:06:17.791587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:06:17.791587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:06:17.791587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 21 01:06:17.791587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:17.791587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 546/1087] 10.222.0.47 () {56 vars in 1120 bytes} [Sun Aug 21 01:06:17 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:17.796881 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98315bbe-db2d-4c85-9ba4-eaeaad848edf None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:17.907089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98315bbe-db2d-4c85-9ba4-eaeaad848edf None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:17.940134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9408cfc7-aa22-4990-bf80-522351a169ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:17.940733 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9408cfc7-aa22-4990-bf80-522351a169ce None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:17.941006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9408cfc7-aa22-4990-bf80-522351a169ce None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:17.941281 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9408cfc7-aa22-4990-bf80-522351a169ce None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:17.941922 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9408cfc7-aa22-4990-bf80-522351a169ce None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:17.942214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9408cfc7-aa22-4990-bf80-522351a169ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:17.943550 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9408cfc7-aa22-4990-bf80-522351a169ce None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:17.951251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 542/1088] 10.222.0.47 () {56 vars in 1115 bytes} [Sun Aug 21 01:06:17 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:17.956831 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28552780-539e-4d7b-b93b-0fdc5fab69a8 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:18.026268 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:18.056876 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28552780-539e-4d7b-b93b-0fdc5fab69a8 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:18.090074 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1d538b-0e4c-4299-bd2c-b876cdefe001 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:18.090700 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1d538b-0e4c-4299-bd2c-b876cdefe001 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:18.090971 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1d538b-0e4c-4299-bd2c-b876cdefe001 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:18.091241 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1d538b-0e4c-4299-bd2c-b876cdefe001 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:18.102668 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1d538b-0e4c-4299-bd2c-b876cdefe001 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:18.104076 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1d538b-0e4c-4299-bd2c-b876cdefe001 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:18.118171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-ce1d538b-0e4c-4299-bd2c-b876cdefe001 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:18.118171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:18.118171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:18.118171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:18.118171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:18.118171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:18.118171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:18.118171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:18.118171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:18.118171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:18.118171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:18.118171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:18.118171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:06:18.118171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:06:18.118171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:06:18.118171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:18.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:06:18.120462 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:06:18.120462 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:18.120462 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:18.120462 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 547/1089] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:18.125143 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9408cfc7-aa22-4990-bf80-522351a169ce None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:18.184181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:18.213092 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d486290a-f490-4415-a67c-f0c06f51fc6f None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:18.213640 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d486290a-f490-4415-a67c-f0c06f51fc6f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:18.213640 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d486290a-f490-4415-a67c-f0c06f51fc6f None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:18.215041 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 548/1090] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:06:18.220411 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324275ed-d20b-4455-a595-4ac7331aa4f1 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:18.220691 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324275ed-d20b-4455-a595-4ac7331aa4f1 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:18.220960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324275ed-d20b-4455-a595-4ac7331aa4f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:18.225792 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9408cfc7-aa22-4990-bf80-522351a169ce None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:18.239289 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-324275ed-d20b-4455-a595-4ac7331aa4f1 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:18.240513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-324275ed-d20b-4455-a595-4ac7331aa4f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:18.261730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3c3052-1679-4839-99a0-7725d8b52536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:18.262331 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c3052-1679-4839-99a0-7725d8b52536 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:18.262605 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c3052-1679-4839-99a0-7725d8b52536 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:18.262871 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c3052-1679-4839-99a0-7725d8b52536 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:18.263499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3c3052-1679-4839-99a0-7725d8b52536 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:18.264319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3c3052-1679-4839-99a0-7725d8b52536 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:18.265660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3c3052-1679-4839-99a0-7725d8b52536 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:18.283213 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 543/1091] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:06:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:18.285232 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3c3052-1679-4839-99a0-7725d8b52536 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:18.308160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 549/1092] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:06:18.385155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3c3052-1679-4839-99a0-7725d8b52536 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:18.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:18.419133 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563f4849-dcef-4855-b690-8949e15566da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:18.419760 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563f4849-dcef-4855-b690-8949e15566da None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:18.420214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563f4849-dcef-4855-b690-8949e15566da None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:18.420456 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563f4849-dcef-4855-b690-8949e15566da None admin] PATH_INFO: `/v3/users/nova` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:18.427260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563f4849-dcef-4855-b690-8949e15566da None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:18.428731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563f4849-dcef-4855-b690-8949e15566da None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:18.441392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-563f4849-dcef-4855-b690-8949e15566da None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:06:18.441392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:18.441392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:18.441392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:18.441392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:18.441392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:18.441392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:18.441392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:18.441392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:18.441392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:18.441392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:18.441392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:18.441392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:06:18.441392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:06:18.441392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:06:18.441392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:18.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:18.443496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:18.443496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:18.443496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:06:18.443496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:06:18.443496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:06:18.443496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:06:18.443496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:06:18.443496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:06:18.443496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 21 01:06:18.443496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:18.443496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 544/1093] 10.222.0.47 () {56 vars in 1093 bytes} [Sun Aug 21 01:06:18 2022] GET /identity/v3/users/nova => generated 82 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:06:18.448278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324275ed-d20b-4455-a595-4ac7331aa4f1 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:18.502041 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:18.545809 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324275ed-d20b-4455-a595-4ac7331aa4f1 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:18.575229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563f4849-dcef-4855-b690-8949e15566da None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:18.579615 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4dfe4a-1d1e-4cf9-8e07-c38cddabf4cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:18.580277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4dfe4a-1d1e-4cf9-8e07-c38cddabf4cd None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:18.580557 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4dfe4a-1d1e-4cf9-8e07-c38cddabf4cd None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:18.580824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4dfe4a-1d1e-4cf9-8e07-c38cddabf4cd None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:18.581540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4dfe4a-1d1e-4cf9-8e07-c38cddabf4cd None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:18.581854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4dfe4a-1d1e-4cf9-8e07-c38cddabf4cd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:18.583264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4dfe4a-1d1e-4cf9-8e07-c38cddabf4cd None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:18.600738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 550/1094] 10.222.0.47 () {56 vars in 1133 bytes} [Sun Aug 21 01:06:18 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:18.606305 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4dfe4a-1d1e-4cf9-8e07-c38cddabf4cd None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:18.677642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563f4849-dcef-4855-b690-8949e15566da None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:18.705719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4dfe4a-1d1e-4cf9-8e07-c38cddabf4cd None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:18.711694 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f408fad-dbb6-4239-b489-cfa38b2a8c76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:18.712347 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f408fad-dbb6-4239-b489-cfa38b2a8c76 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:18.712620 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f408fad-dbb6-4239-b489-cfa38b2a8c76 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:18.712884 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f408fad-dbb6-4239-b489-cfa38b2a8c76 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:18.719155 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:18.723621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f408fad-dbb6-4239-b489-cfa38b2a8c76 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:06:18.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-3f408fad-dbb6-4239-b489-cfa38b2a8c76 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:06:18.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:18.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:18.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:18.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:18.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:18.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:18.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:18.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:18.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:18.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:18.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:18.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:06:18.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:06:18.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:06:18.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:06:18.729294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:06:18.729294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:06:18.729294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:06:18.729294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:06:18.729294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:18.729294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:18.729294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:18.729294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:18.729294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:18.729294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:18.729294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:18.729294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:18.729294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:18.729294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:18.729294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:18.729294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:18.730170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:18.730170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:18.730170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:18.730170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:18.730170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:06:18.730170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:06:18.730170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:06:18.730170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:06:18.730170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:06:18.730170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:06:18.730170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:06:18.730170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:18.730170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 545/1095] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:06:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:06:18.735404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f408fad-dbb6-4239-b489-cfa38b2a8c76 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:18.740596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd156f7-025d-43e1-beaf-31730e7a240b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:18.742857 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd156f7-025d-43e1-beaf-31730e7a240b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:18.742857 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd156f7-025d-43e1-beaf-31730e7a240b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:18.742857 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd156f7-025d-43e1-beaf-31730e7a240b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:18.778993 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:18.809290 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd156f7-025d-43e1-beaf-31730e7a240b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:18.811152 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd156f7-025d-43e1-beaf-31730e7a240b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:18.836407 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f408fad-dbb6-4239-b489-cfa38b2a8c76 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:18.870450 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ac4a9c-25cc-4959-901e-ee327f83ff3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:18.871055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ac4a9c-25cc-4959-901e-ee327f83ff3e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:18.871326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ac4a9c-25cc-4959-901e-ee327f83ff3e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:18.871596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ac4a9c-25cc-4959-901e-ee327f83ff3e None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:18.872273 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ac4a9c-25cc-4959-901e-ee327f83ff3e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:18.872569 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ac4a9c-25cc-4959-901e-ee327f83ff3e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:18.873981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ac4a9c-25cc-4959-901e-ee327f83ff3e None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:18.881815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 546/1096] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:06:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:18.887503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ac4a9c-25cc-4959-901e-ee327f83ff3e None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:18.912433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 551/1097] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:06:18 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 21 01:06:18.918029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd156f7-025d-43e1-beaf-31730e7a240b None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:18.931828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:18.987665 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ac4a9c-25cc-4959-901e-ee327f83ff3e None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:18.989217 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:19.015912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd156f7-025d-43e1-beaf-31730e7a240b None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:19.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57a9e4c-c05b-4d6c-af09-9e1d9880a804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:19.022989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57a9e4c-c05b-4d6c-af09-9e1d9880a804 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:19.023268 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57a9e4c-c05b-4d6c-af09-9e1d9880a804 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:19.023540 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57a9e4c-c05b-4d6c-af09-9e1d9880a804 None admin] PATH_INFO: `/v3/users/58d95df9c2014d63aefb9311e852e828` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:19.024696 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57a9e4c-c05b-4d6c-af09-9e1d9880a804 None admin] RBAC: Authorizing `identity:get_user(user_id=58d95df9c2014d63aefb9311e852e828)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:19.026118 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57a9e4c-c05b-4d6c-af09-9e1d9880a804 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:19.027866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 547/1098] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:06:18 2022] GET /identity/v3/users/58d95df9c2014d63aefb9311e852e828 => generated 285 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:19.034097 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57a9e4c-c05b-4d6c-af09-9e1d9880a804 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:19.054506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ba2892-7981-4319-b5d4-5825b1b77d09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:19.055143 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ba2892-7981-4319-b5d4-5825b1b77d09 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:19.055431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ba2892-7981-4319-b5d4-5825b1b77d09 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:19.055704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ba2892-7981-4319-b5d4-5825b1b77d09 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:19.066235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ba2892-7981-4319-b5d4-5825b1b77d09 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:19.067593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ba2892-7981-4319-b5d4-5825b1b77d09 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:19.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-88ba2892-7981-4319-b5d4-5825b1b77d09 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:19.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:19.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:19.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:19.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:19.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:19.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:19.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:19.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:19.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:19.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:19.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:19.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:06:19.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:06:19.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:06:19.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:19.083879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:06:19.084873 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:06:19.084873 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:06:19.084873 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:19.084873 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 552/1099] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:19.089492 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ba2892-7981-4319-b5d4-5825b1b77d09 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:19.132070 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57a9e4c-c05b-4d6c-af09-9e1d9880a804 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:19.150676 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:19.168706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3838c99-d310-4cba-abfd-2f532bbffcb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:19.169304 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3838c99-d310-4cba-abfd-2f532bbffcb3 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:19.169582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3838c99-d310-4cba-abfd-2f532bbffcb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:19.169876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3838c99-d310-4cba-abfd-2f532bbffcb3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:19.188446 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ba2892-7981-4319-b5d4-5825b1b77d09 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:19.209446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:19.221941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029982a5-3774-4e3e-a4c6-2cdf1c2a9a4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:19.222533 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029982a5-3774-4e3e-a4c6-2cdf1c2a9a4e None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:19.223132 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029982a5-3774-4e3e-a4c6-2cdf1c2a9a4e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:19.223132 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029982a5-3774-4e3e-a4c6-2cdf1c2a9a4e None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:19.223681 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029982a5-3774-4e3e-a4c6-2cdf1c2a9a4e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:19.224067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029982a5-3774-4e3e-a4c6-2cdf1c2a9a4e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:19.225416 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029982a5-3774-4e3e-a4c6-2cdf1c2a9a4e None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:19.237016 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 553/1100] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:06:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:19.241667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3838c99-d310-4cba-abfd-2f532bbffcb3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:19.241667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3838c99-d310-4cba-abfd-2f532bbffcb3 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:19.242537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029982a5-3774-4e3e-a4c6-2cdf1c2a9a4e None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:19.339364 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 548/1101] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:06:19 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 21 01:06:19.340425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029982a5-3774-4e3e-a4c6-2cdf1c2a9a4e None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:19.345487 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3838c99-d310-4cba-abfd-2f532bbffcb3 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:19.370097 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:19.381967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410145d1-1ea5-4fd1-a79a-94d466a60e41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:19.382588 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410145d1-1ea5-4fd1-a79a-94d466a60e41 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:19.382872 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410145d1-1ea5-4fd1-a79a-94d466a60e41 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:19.383140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410145d1-1ea5-4fd1-a79a-94d466a60e41 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:19.390369 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410145d1-1ea5-4fd1-a79a-94d466a60e41 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:19.391792 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410145d1-1ea5-4fd1-a79a-94d466a60e41 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:19.403722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-410145d1-1ea5-4fd1-a79a-94d466a60e41 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:06:19.403722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:19.403722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:19.403722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:19.403722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:19.403722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:19.403722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:19.403722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:19.403722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:19.403722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:19.403722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:19.403722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:19.403722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:06:19.403722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:06:19.403722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:19.404709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:19.405801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:06:19.405801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:06:19.405801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:06:19.405801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:06:19.405801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:06:19.405801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:06:19.405801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:06:19.405801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:19.405801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 554/1102] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:06:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:19.411410 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410145d1-1ea5-4fd1-a79a-94d466a60e41 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:19.413397 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:19.444126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3838c99-d310-4cba-abfd-2f532bbffcb3 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:19.478531 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff184210-8e1c-471f-aeb6-f74364e62c43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:19.479130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff184210-8e1c-471f-aeb6-f74364e62c43 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:19.479405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff184210-8e1c-471f-aeb6-f74364e62c43 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:19.479677 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff184210-8e1c-471f-aeb6-f74364e62c43 None admin] PATH_INFO: `/v3/projects/ef43bd38583d4e5794474db52b2f8958` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:19.480812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff184210-8e1c-471f-aeb6-f74364e62c43 None admin] RBAC: Authorizing `identity:get_project(project_id=ef43bd38583d4e5794474db52b2f8958)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:19.482229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff184210-8e1c-471f-aeb6-f74364e62c43 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:19.483859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 549/1103] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:06:19 2022] GET /identity/v3/projects/ef43bd38583d4e5794474db52b2f8958 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:19.489772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff184210-8e1c-471f-aeb6-f74364e62c43 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:19.510369 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410145d1-1ea5-4fd1-a79a-94d466a60e41 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:19.543731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621d07ee-8ce7-4aef-a04b-995cf223cfbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:19.544361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621d07ee-8ce7-4aef-a04b-995cf223cfbd None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:19.544633 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621d07ee-8ce7-4aef-a04b-995cf223cfbd None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:19.544908 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621d07ee-8ce7-4aef-a04b-995cf223cfbd None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:19.545582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621d07ee-8ce7-4aef-a04b-995cf223cfbd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:19.545881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621d07ee-8ce7-4aef-a04b-995cf223cfbd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:19.547291 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621d07ee-8ce7-4aef-a04b-995cf223cfbd None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:19.558785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 555/1104] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:06:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:19.565551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621d07ee-8ce7-4aef-a04b-995cf223cfbd None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:19.589021 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff184210-8e1c-471f-aeb6-f74364e62c43 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:19.610201 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:19.621983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad139e44-4bc8-42fa-b2c4-ad41503d5a03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:19.622580 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad139e44-4bc8-42fa-b2c4-ad41503d5a03 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:19.623379 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad139e44-4bc8-42fa-b2c4-ad41503d5a03 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:19.623649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad139e44-4bc8-42fa-b2c4-ad41503d5a03 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:19.624420 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad139e44-4bc8-42fa-b2c4-ad41503d5a03 None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=ef43bd38583d4e5794474db52b2f8958, user.id=58d95df9c2014d63aefb9311e852e828) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:19.624721 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad139e44-4bc8-42fa-b2c4-ad41503d5a03 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:19.626177 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad139e44-4bc8-42fa-b2c4-ad41503d5a03 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:19.631814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 550/1105] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:06:19 2022] GET /identity/v3/role_assignments?user.id=58d95df9c2014d63aefb9311e852e828&scope.project.id=ef43bd38583d4e5794474db52b2f8958&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:19.666790 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621d07ee-8ce7-4aef-a04b-995cf223cfbd None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:19.708439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d4a005-c054-4743-8f77-e538e178d0a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:19.709062 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d4a005-c054-4743-8f77-e538e178d0a0 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:19.709334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d4a005-c054-4743-8f77-e538e178d0a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:19.709603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d4a005-c054-4743-8f77-e538e178d0a0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:19.710337 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d4a005-c054-4743-8f77-e538e178d0a0 None admin] RBAC: Adding query filter params (role.id=830cf269498649038ec8a27a3712fc60, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=fabdcdc25d1d4ef6bc66bd6cadecaaf8) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:19.710644 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d4a005-c054-4743-8f77-e538e178d0a0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:19.712175 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d4a005-c054-4743-8f77-e538e178d0a0 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:19.718611 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 556/1106] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:06:19 2022] GET /identity/v3/role_assignments?user.id=fabdcdc25d1d4ef6bc66bd6cadecaaf8&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=830cf269498649038ec8a27a3712fc60 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:21.598779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17491da1-d62d-4240-8d88-44493b44473c None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:21.599327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17491da1-d62d-4240-8d88-44493b44473c None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:21.599327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17491da1-d62d-4240-8d88-44493b44473c None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:21.600608 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 551/1107] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:21.605811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b47d9f4-7b82-4a9b-9aa5-5c2b3b703f8c None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:21.606263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b47d9f4-7b82-4a9b-9aa5-5c2b3b703f8c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:21.606368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b47d9f4-7b82-4a9b-9aa5-5c2b3b703f8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:21.624921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-9b47d9f4-7b82-4a9b-9aa5-5c2b3b703f8c None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:21.626082 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9b47d9f4-7b82-4a9b-9aa5-5c2b3b703f8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:21.690339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 557/1108] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:06:21.961393 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17491da1-d62d-4240-8d88-44493b44473c None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:21.964368 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:22.062374 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17491da1-d62d-4240-8d88-44493b44473c None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:22.095730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9a4093-a395-4fa3-ba4f-2c7d300b6e15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:22.096369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9a4093-a395-4fa3-ba4f-2c7d300b6e15 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:22.096647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9a4093-a395-4fa3-ba4f-2c7d300b6e15 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:22.096915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9a4093-a395-4fa3-ba4f-2c7d300b6e15 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:22.098005 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9a4093-a395-4fa3-ba4f-2c7d300b6e15 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:22.099421 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9a4093-a395-4fa3-ba4f-2c7d300b6e15 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:22.101013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 552/1109] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:22.108889 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b47d9f4-7b82-4a9b-9aa5-5c2b3b703f8c None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:22.146812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:22.204339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b47d9f4-7b82-4a9b-9aa5-5c2b3b703f8c None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:22.237226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab03f34-8b51-4e2e-bdc7-e2977e575cc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:22.237763 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab03f34-8b51-4e2e-bdc7-e2977e575cc9 None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:22.238039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab03f34-8b51-4e2e-bdc7-e2977e575cc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:22.238308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab03f34-8b51-4e2e-bdc7-e2977e575cc9 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:22.238939 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab03f34-8b51-4e2e-bdc7-e2977e575cc9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:22.240452 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab03f34-8b51-4e2e-bdc7-e2977e575cc9 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:22.347176 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 558/1110] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:06:22 2022] POST /identity/v3/users => generated 286 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:06:24.284703 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c65eafb-78d2-479c-9483-9642750ab715 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:24.285261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c65eafb-78d2-479c-9483-9642750ab715 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:24.285261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c65eafb-78d2-479c-9483-9642750ab715 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:24.286580 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 553/1111] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:24.291822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c995b7-1233-49f3-a78d-32583bae6c86 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:24.292283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c995b7-1233-49f3-a78d-32583bae6c86 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:24.292383 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c995b7-1233-49f3-a78d-32583bae6c86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:24.311234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f9c995b7-1233-49f3-a78d-32583bae6c86 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:24.312446 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f9c995b7-1233-49f3-a78d-32583bae6c86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:24.377449 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 559/1112] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:24.641325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c65eafb-78d2-479c-9483-9642750ab715 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:24.710580 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:24.743365 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c65eafb-78d2-479c-9483-9642750ab715 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:24.776029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11054ea9-e0d3-4bec-9480-e5f1b0e11f78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:24.776682 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11054ea9-e0d3-4bec-9480-e5f1b0e11f78 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:24.776958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11054ea9-e0d3-4bec-9480-e5f1b0e11f78 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:24.777229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11054ea9-e0d3-4bec-9480-e5f1b0e11f78 None admin] PATH_INFO: `/v3/roles/fbc5961470b74a85bde6be2171f5f08f` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:24.779133 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11054ea9-e0d3-4bec-9480-e5f1b0e11f78 None admin] RBAC: Authorizing `identity:get_role(role_id=fbc5961470b74a85bde6be2171f5f08f)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:24.780519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11054ea9-e0d3-4bec-9480-e5f1b0e11f78 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:24.781847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 554/1113] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:06:24 2022] GET /identity/v3/roles/fbc5961470b74a85bde6be2171f5f08f => generated 223 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:06:24.787791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c995b7-1233-49f3-a78d-32583bae6c86 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:24.811960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:24.886097 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c995b7-1233-49f3-a78d-32583bae6c86 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:24.918778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d4e21d-d942-465a-9583-26342b5042e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:24.919892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d4e21d-d942-465a-9583-26342b5042e1 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:24.920192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d4e21d-d942-465a-9583-26342b5042e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:24.920470 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d4e21d-d942-465a-9583-26342b5042e1 None admin] PATH_INFO: `/v3/users/8fc3aad1930c40e3983f0ffa2f4ce64e` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:24.932912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d4e21d-d942-465a-9583-26342b5042e1 None admin] RBAC: Authorizing `identity:get_user(user_id=8fc3aad1930c40e3983f0ffa2f4ce64e)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:24.934327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d4e21d-d942-465a-9583-26342b5042e1 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:24.935936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 560/1114] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:06:24 2022] GET /identity/v3/users/8fc3aad1930c40e3983f0ffa2f4ce64e => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:24.941924 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11054ea9-e0d3-4bec-9480-e5f1b0e11f78 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:25.038757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11054ea9-e0d3-4bec-9480-e5f1b0e11f78 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:25.072824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1381a6-ead1-4b3b-ad64-28db1c8c9cb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:25.073417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1381a6-ead1-4b3b-ad64-28db1c8c9cb6 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:25.073691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1381a6-ead1-4b3b-ad64-28db1c8c9cb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:25.073957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1381a6-ead1-4b3b-ad64-28db1c8c9cb6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:25.083296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:25.141704 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1381a6-ead1-4b3b-ad64-28db1c8c9cb6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:25.143155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1381a6-ead1-4b3b-ad64-28db1c8c9cb6 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:25.241028 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 555/1115] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:06:24 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:06:25.247223 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d4e21d-d942-465a-9583-26342b5042e1 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:25.329956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:25.348182 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d4e21d-d942-465a-9583-26342b5042e1 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:25.383728 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455d10a7-85f1-45c6-8dc1-ee87a0acd749 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:25.384394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455d10a7-85f1-45c6-8dc1-ee87a0acd749 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:25.384676 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455d10a7-85f1-45c6-8dc1-ee87a0acd749 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:25.384949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455d10a7-85f1-45c6-8dc1-ee87a0acd749 None admin] PATH_INFO: `/v3/projects/ef43bd38583d4e5794474db52b2f8958` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:25.386068 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455d10a7-85f1-45c6-8dc1-ee87a0acd749 None admin] RBAC: Authorizing `identity:get_project(project_id=ef43bd38583d4e5794474db52b2f8958)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:25.387488 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455d10a7-85f1-45c6-8dc1-ee87a0acd749 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:25.389828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 561/1116] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:06:25 2022] GET /identity/v3/projects/ef43bd38583d4e5794474db52b2f8958 => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:25.396250 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1381a6-ead1-4b3b-ad64-28db1c8c9cb6 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:25.435658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:25.496369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1381a6-ead1-4b3b-ad64-28db1c8c9cb6 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:25.531496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdce68a8-c636-42b7-be8a-59a4703d1679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:25.532087 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdce68a8-c636-42b7-be8a-59a4703d1679 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:25.532389 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdce68a8-c636-42b7-be8a-59a4703d1679 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:25.532653 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdce68a8-c636-42b7-be8a-59a4703d1679 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:25.533363 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdce68a8-c636-42b7-be8a-59a4703d1679 None admin] RBAC: Adding query filter params (role.id=fbc5961470b74a85bde6be2171f5f08f, scope.project.id=ef43bd38583d4e5794474db52b2f8958, user.id=8fc3aad1930c40e3983f0ffa2f4ce64e) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:25.533659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdce68a8-c636-42b7-be8a-59a4703d1679 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:25.535144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdce68a8-c636-42b7-be8a-59a4703d1679 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:25.540565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 556/1117] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:06:25 2022] GET /identity/v3/role_assignments?user.id=8fc3aad1930c40e3983f0ffa2f4ce64e&scope.project.id=ef43bd38583d4e5794474db52b2f8958&role.id=fbc5961470b74a85bde6be2171f5f08f => generated 259 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:27.417840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c86ac75-74e0-48ec-8ee6-1fec5b0c88c2 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:27.418349 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c86ac75-74e0-48ec-8ee6-1fec5b0c88c2 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:27.418349 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c86ac75-74e0-48ec-8ee6-1fec5b0c88c2 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:27.419790 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 562/1118] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:27.425038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c33c38-e706-4b0f-8d7f-1c0893d5a461 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:27.425454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c33c38-e706-4b0f-8d7f-1c0893d5a461 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:27.425582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c33c38-e706-4b0f-8d7f-1c0893d5a461 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:27.443622 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-29c33c38-e706-4b0f-8d7f-1c0893d5a461 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:27.444827 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-29c33c38-e706-4b0f-8d7f-1c0893d5a461 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:27.510055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 557/1119] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:27.783638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c86ac75-74e0-48ec-8ee6-1fec5b0c88c2 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:27.881693 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c86ac75-74e0-48ec-8ee6-1fec5b0c88c2 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:27.914601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cdb984-e697-4417-9948-54ea9bbb0c51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:27.915215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cdb984-e697-4417-9948-54ea9bbb0c51 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:27.915491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cdb984-e697-4417-9948-54ea9bbb0c51 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:27.915760 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cdb984-e697-4417-9948-54ea9bbb0c51 None admin] PATH_INFO: `/v3/roles/fbc5961470b74a85bde6be2171f5f08f` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:27.917158 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cdb984-e697-4417-9948-54ea9bbb0c51 None admin] RBAC: Authorizing `identity:get_role(role_id=fbc5961470b74a85bde6be2171f5f08f)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:27.918501 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cdb984-e697-4417-9948-54ea9bbb0c51 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:27.919832 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 563/1120] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:06:27 2022] GET /identity/v3/roles/fbc5961470b74a85bde6be2171f5f08f => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:27.925672 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c33c38-e706-4b0f-8d7f-1c0893d5a461 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:27.956499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:28.021196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c33c38-e706-4b0f-8d7f-1c0893d5a461 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:28.054977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eeb60c8-d3c3-4506-96d8-63abc3438714 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:28.055534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eeb60c8-d3c3-4506-96d8-63abc3438714 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:28.055806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eeb60c8-d3c3-4506-96d8-63abc3438714 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:28.056072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eeb60c8-d3c3-4506-96d8-63abc3438714 None admin] PATH_INFO: `/v3/users/8fc3aad1930c40e3983f0ffa2f4ce64e` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:28.057201 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eeb60c8-d3c3-4506-96d8-63abc3438714 None admin] RBAC: Authorizing `identity:get_user(user_id=8fc3aad1930c40e3983f0ffa2f4ce64e)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:28.058604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eeb60c8-d3c3-4506-96d8-63abc3438714 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:28.060793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 558/1121] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:06:27 2022] GET /identity/v3/users/8fc3aad1930c40e3983f0ffa2f4ce64e => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:28.066855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cdb984-e697-4417-9948-54ea9bbb0c51 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:28.088528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:28.171318 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cdb984-e697-4417-9948-54ea9bbb0c51 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:28.207581 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479cb5c7-982a-4bb0-842d-61e5c7c6535d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:28.208234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479cb5c7-982a-4bb0-842d-61e5c7c6535d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:28.208514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479cb5c7-982a-4bb0-842d-61e5c7c6535d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:28.208784 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479cb5c7-982a-4bb0-842d-61e5c7c6535d None admin] PATH_INFO: `/v3/projects/ef43bd38583d4e5794474db52b2f8958` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:28.209892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479cb5c7-982a-4bb0-842d-61e5c7c6535d None admin] RBAC: Authorizing `identity:get_project(project_id=ef43bd38583d4e5794474db52b2f8958)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:28.211301 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479cb5c7-982a-4bb0-842d-61e5c7c6535d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:28.212955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 564/1122] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:06:28 2022] GET /identity/v3/projects/ef43bd38583d4e5794474db52b2f8958 => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:28.218967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eeb60c8-d3c3-4506-96d8-63abc3438714 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:28.313031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:28.316091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eeb60c8-d3c3-4506-96d8-63abc3438714 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:28.349809 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332e63f4-9aa2-4d79-9b9d-4d2341006c50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:28.350452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332e63f4-9aa2-4d79-9b9d-4d2341006c50 None admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:28.350725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332e63f4-9aa2-4d79-9b9d-4d2341006c50 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:28.350993 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332e63f4-9aa2-4d79-9b9d-4d2341006c50 None admin] PATH_INFO: `/v3/projects/ef43bd38583d4e5794474db52b2f8958/users/8fc3aad1930c40e3983f0ffa2f4ce64e/roles/fbc5961470b74a85bde6be2171f5f08f` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:28.353119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332e63f4-9aa2-4d79-9b9d-4d2341006c50 None admin] RBAC: Authorizing `identity:create_grant(project_id=ef43bd38583d4e5794474db52b2f8958, user_id=8fc3aad1930c40e3983f0ffa2f4ce64e, role_id=fbc5961470b74a85bde6be2171f5f08f)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:28.354530 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332e63f4-9aa2-4d79-9b9d-4d2341006c50 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:28.432779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 559/1123] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:06:28 2022] PUT /identity/v3/projects/ef43bd38583d4e5794474db52b2f8958/users/8fc3aad1930c40e3983f0ffa2f4ce64e/roles/fbc5961470b74a85bde6be2171f5f08f => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:06:30.291174 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ca564f-4e06-400e-a687-94484ff702e4 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:30.291732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ca564f-4e06-400e-a687-94484ff702e4 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:30.291732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ca564f-4e06-400e-a687-94484ff702e4 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:30.293103 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 565/1124] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:30.298895 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece0c94c-4dd1-4eb5-a13a-4f9534c6dfc2 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:30.299275 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece0c94c-4dd1-4eb5-a13a-4f9534c6dfc2 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:30.299437 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece0c94c-4dd1-4eb5-a13a-4f9534c6dfc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:30.317934 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-ece0c94c-4dd1-4eb5-a13a-4f9534c6dfc2 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:30.319113 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ece0c94c-4dd1-4eb5-a13a-4f9534c6dfc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:30.386874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 560/1125] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:06:30.652382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ca564f-4e06-400e-a687-94484ff702e4 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:30.736354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:30.751286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ca564f-4e06-400e-a687-94484ff702e4 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:30.784544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90875c0-885b-4c60-9878-8def435c1458 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:30.785162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90875c0-885b-4c60-9878-8def435c1458 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:30.785436 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90875c0-885b-4c60-9878-8def435c1458 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:30.785703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90875c0-885b-4c60-9878-8def435c1458 None admin] PATH_INFO: `/v3/roles/fbc5961470b74a85bde6be2171f5f08f` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:30.787063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90875c0-885b-4c60-9878-8def435c1458 None admin] RBAC: Authorizing `identity:get_role(role_id=fbc5961470b74a85bde6be2171f5f08f)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:30.788419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90875c0-885b-4c60-9878-8def435c1458 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:30.789717 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 566/1126] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:06:30 2022] GET /identity/v3/roles/fbc5961470b74a85bde6be2171f5f08f => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:30.796080 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece0c94c-4dd1-4eb5-a13a-4f9534c6dfc2 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:30.883028 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:30.891896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece0c94c-4dd1-4eb5-a13a-4f9534c6dfc2 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:30.925433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce319998-97ed-43eb-850f-da248a38f18c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:30.925988 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce319998-97ed-43eb-850f-da248a38f18c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:30.926262 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce319998-97ed-43eb-850f-da248a38f18c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:30.926529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce319998-97ed-43eb-850f-da248a38f18c None admin] PATH_INFO: `/v3/users/8fc3aad1930c40e3983f0ffa2f4ce64e` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:30.927646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce319998-97ed-43eb-850f-da248a38f18c None admin] RBAC: Authorizing `identity:get_user(user_id=8fc3aad1930c40e3983f0ffa2f4ce64e)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:30.929093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce319998-97ed-43eb-850f-da248a38f18c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:30.930666 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 561/1127] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:06:30 2022] GET /identity/v3/users/8fc3aad1930c40e3983f0ffa2f4ce64e => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:30.936349 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90875c0-885b-4c60-9878-8def435c1458 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:31.033419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90875c0-885b-4c60-9878-8def435c1458 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:31.066290 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff21ae8-d9eb-4ab4-9d63-289ca5820fc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:31.066889 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff21ae8-d9eb-4ab4-9d63-289ca5820fc3 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:31.067165 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff21ae8-d9eb-4ab4-9d63-289ca5820fc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:31.067475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff21ae8-d9eb-4ab4-9d63-289ca5820fc3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:31.091659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:31.133424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff21ae8-d9eb-4ab4-9d63-289ca5820fc3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:31.134829 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff21ae8-d9eb-4ab4-9d63-289ca5820fc3 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:31.229348 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 567/1128] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:06:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:06:31.235237 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce319998-97ed-43eb-850f-da248a38f18c None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:31.332023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce319998-97ed-43eb-850f-da248a38f18c None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:31.365498 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01f5019-11a5-48f3-b405-b192f933a6bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:31.366122 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01f5019-11a5-48f3-b405-b192f933a6bb None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:31.366393 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01f5019-11a5-48f3-b405-b192f933a6bb None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:31.366659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01f5019-11a5-48f3-b405-b192f933a6bb None admin] PATH_INFO: `/v3/projects/ef43bd38583d4e5794474db52b2f8958` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:31.367753 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01f5019-11a5-48f3-b405-b192f933a6bb None admin] RBAC: Authorizing `identity:get_project(project_id=ef43bd38583d4e5794474db52b2f8958)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:31.369195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01f5019-11a5-48f3-b405-b192f933a6bb None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:31.370813 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 562/1129] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:06:31 2022] GET /identity/v3/projects/ef43bd38583d4e5794474db52b2f8958 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:31.377478 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff21ae8-d9eb-4ab4-9d63-289ca5820fc3 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:31.429743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:31.473651 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff21ae8-d9eb-4ab4-9d63-289ca5820fc3 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:31.505469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a51bc00-2402-4f0a-8bde-4c2850f017f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:31.506003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a51bc00-2402-4f0a-8bde-4c2850f017f9 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:31.506275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a51bc00-2402-4f0a-8bde-4c2850f017f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:31.506542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a51bc00-2402-4f0a-8bde-4c2850f017f9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:31.507242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a51bc00-2402-4f0a-8bde-4c2850f017f9 None admin] RBAC: Adding query filter params (role.id=fbc5961470b74a85bde6be2171f5f08f, scope.project.id=ef43bd38583d4e5794474db52b2f8958, user.id=8fc3aad1930c40e3983f0ffa2f4ce64e) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:31.507540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a51bc00-2402-4f0a-8bde-4c2850f017f9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:31.508963 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a51bc00-2402-4f0a-8bde-4c2850f017f9 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:31.514992 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 568/1130] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:06:31 2022] GET /identity/v3/role_assignments?user.id=8fc3aad1930c40e3983f0ffa2f4ce64e&scope.project.id=ef43bd38583d4e5794474db52b2f8958&role.id=fbc5961470b74a85bde6be2171f5f08f => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:33.434528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b9d055-1487-421c-8986-78f91ae56d4f None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:33.435072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b9d055-1487-421c-8986-78f91ae56d4f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:33.435072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b9d055-1487-421c-8986-78f91ae56d4f None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:33.436440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 563/1131] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:33.441985 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb80850-8e20-45d0-a23a-f72f6e796296 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:33.442430 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb80850-8e20-45d0-a23a-f72f6e796296 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:33.442533 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb80850-8e20-45d0-a23a-f72f6e796296 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:33.459918 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-2fb80850-8e20-45d0-a23a-f72f6e796296 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:33.461122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-2fb80850-8e20-45d0-a23a-f72f6e796296 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:33.525211 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 569/1132] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:33.795576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b9d055-1487-421c-8986-78f91ae56d4f None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:33.822879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:33.896172 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b9d055-1487-421c-8986-78f91ae56d4f None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:33.928538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93925dd-ec60-4f8f-811c-a3584f9e12fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:33.929155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93925dd-ec60-4f8f-811c-a3584f9e12fb None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:33.929429 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93925dd-ec60-4f8f-811c-a3584f9e12fb None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:33.929696 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93925dd-ec60-4f8f-811c-a3584f9e12fb None admin] PATH_INFO: `/v3/domains/default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:33.930791 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93925dd-ec60-4f8f-811c-a3584f9e12fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:33.932367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93925dd-ec60-4f8f-811c-a3584f9e12fb None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:33.934422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 564/1133] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:33.940423 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb80850-8e20-45d0-a23a-f72f6e796296 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:33.986924 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:34.035825 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb80850-8e20-45d0-a23a-f72f6e796296 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:34.069033 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82ebbb1-d862-4a9f-a284-59ce2d4c6e4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:34.069564 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82ebbb1-d862-4a9f-a284-59ce2d4c6e4b None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:34.069841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82ebbb1-d862-4a9f-a284-59ce2d4c6e4b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:34.070110 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82ebbb1-d862-4a9f-a284-59ce2d4c6e4b None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:34.070745 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82ebbb1-d862-4a9f-a284-59ce2d4c6e4b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:34.072172 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82ebbb1-d862-4a9f-a284-59ce2d4c6e4b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:34.169932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 570/1134] 10.222.0.47 () {60 vars in 1141 bytes} [Sun Aug 21 01:06:33 2022] POST /identity/v3/projects => generated 311 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:06:36.108953 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7782ec11-c1fd-44f8-bf14-6449a7a8abe5 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:36.109512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7782ec11-c1fd-44f8-bf14-6449a7a8abe5 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:36.109512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7782ec11-c1fd-44f8-bf14-6449a7a8abe5 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:36.110859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 565/1135] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:36.116116 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6c1cf5-3c71-4cd4-a114-79c45bb91e62 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:36.116551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6c1cf5-3c71-4cd4-a114-79c45bb91e62 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:36.116684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6c1cf5-3c71-4cd4-a114-79c45bb91e62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:36.139139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-fa6c1cf5-3c71-4cd4-a114-79c45bb91e62 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:36.140360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-fa6c1cf5-3c71-4cd4-a114-79c45bb91e62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:36.205569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 571/1136] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:36.475353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7782ec11-c1fd-44f8-bf14-6449a7a8abe5 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:36.567984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:36.574986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7782ec11-c1fd-44f8-bf14-6449a7a8abe5 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:36.607986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df797e3-1e85-476a-90b2-9ad6ef84967f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:36.608631 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df797e3-1e85-476a-90b2-9ad6ef84967f None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:36.608908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df797e3-1e85-476a-90b2-9ad6ef84967f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:36.609176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df797e3-1e85-476a-90b2-9ad6ef84967f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:36.610269 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df797e3-1e85-476a-90b2-9ad6ef84967f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:36.611692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df797e3-1e85-476a-90b2-9ad6ef84967f None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:36.613275 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 566/1137] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:06:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:36.619006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6c1cf5-3c71-4cd4-a114-79c45bb91e62 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:36.648215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:36.717287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6c1cf5-3c71-4cd4-a114-79c45bb91e62 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:36.750292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8994cb26-6ca4-4a10-beb7-ca9f81286973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:36.750834 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8994cb26-6ca4-4a10-beb7-ca9f81286973 None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:36.751112 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8994cb26-6ca4-4a10-beb7-ca9f81286973 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:36.751383 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8994cb26-6ca4-4a10-beb7-ca9f81286973 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:36.752022 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8994cb26-6ca4-4a10-beb7-ca9f81286973 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:36.753454 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8994cb26-6ca4-4a10-beb7-ca9f81286973 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:36.870867 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 572/1138] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:06:36 2022] POST /identity/v3/users => generated 286 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:06:38.786486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997426c2-22ad-43c5-a01d-407702741085 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:38.787016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997426c2-22ad-43c5-a01d-407702741085 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:38.787016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997426c2-22ad-43c5-a01d-407702741085 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:38.788445 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 567/1139] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:38.793722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0907d0a7-b657-40c6-8b03-274bb848ae78 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:38.794089 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0907d0a7-b657-40c6-8b03-274bb848ae78 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:38.794267 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0907d0a7-b657-40c6-8b03-274bb848ae78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:38.811810 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-0907d0a7-b657-40c6-8b03-274bb848ae78 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:38.813290 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-0907d0a7-b657-40c6-8b03-274bb848ae78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:38.877961 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 573/1140] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:39.143476 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997426c2-22ad-43c5-a01d-407702741085 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:39.244174 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997426c2-22ad-43c5-a01d-407702741085 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:39.277634 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2212ca-f90a-466c-aea2-adffb9398bee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:39.278248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2212ca-f90a-466c-aea2-adffb9398bee None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:39.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2212ca-f90a-466c-aea2-adffb9398bee None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:39.278795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2212ca-f90a-466c-aea2-adffb9398bee None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:39.288657 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2212ca-f90a-466c-aea2-adffb9398bee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:06:39.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-3d2212ca-f90a-466c-aea2-adffb9398bee None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:06:39.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:39.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:39.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:39.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:39.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:39.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:39.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:39.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:39.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:39.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:39.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:39.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:06:39.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:06:39.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:06:39.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:06:39.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:06:39.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:06:39.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:06:39.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:06:39.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:39.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:39.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:39.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:39.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:39.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:39.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:39.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:39.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:39.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:39.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:39.295041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:39.295962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:39.295962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:39.295962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:39.295962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:39.295962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:06:39.295962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:06:39.295962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:06:39.295962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:06:39.295962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:06:39.295962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:06:39.295962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:06:39.295962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:39.295962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 568/1141] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:06:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:39.301041 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0907d0a7-b657-40c6-8b03-274bb848ae78 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:39.318823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:39.399227 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0907d0a7-b657-40c6-8b03-274bb848ae78 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:39.432160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878fc961-5b56-4399-b20c-1fe92b9c2db9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:39.432757 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878fc961-5b56-4399-b20c-1fe92b9c2db9 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:39.433036 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878fc961-5b56-4399-b20c-1fe92b9c2db9 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:39.433866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878fc961-5b56-4399-b20c-1fe92b9c2db9 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:39.434517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878fc961-5b56-4399-b20c-1fe92b9c2db9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:39.434818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878fc961-5b56-4399-b20c-1fe92b9c2db9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:39.436185 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878fc961-5b56-4399-b20c-1fe92b9c2db9 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:39.444329 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 574/1142] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:06:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:39.450274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2212ca-f90a-466c-aea2-adffb9398bee None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:39.474299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:39.548782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2212ca-f90a-466c-aea2-adffb9398bee None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:39.582007 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab75179-2f82-40ef-8697-ab2dae64d579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:39.582618 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab75179-2f82-40ef-8697-ab2dae64d579 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:39.582892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab75179-2f82-40ef-8697-ab2dae64d579 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:39.583160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab75179-2f82-40ef-8697-ab2dae64d579 None admin] PATH_INFO: `/v3/users/d72058ab2d614b3f96e68e1428cfd45e` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:39.595827 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab75179-2f82-40ef-8697-ab2dae64d579 None admin] RBAC: Authorizing `identity:get_user(user_id=d72058ab2d614b3f96e68e1428cfd45e)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:39.597275 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab75179-2f82-40ef-8697-ab2dae64d579 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:39.598858 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 569/1143] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:06:39 2022] GET /identity/v3/users/d72058ab2d614b3f96e68e1428cfd45e => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:39.604798 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878fc961-5b56-4399-b20c-1fe92b9c2db9 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:39.681843 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:39.702201 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878fc961-5b56-4399-b20c-1fe92b9c2db9 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:39.735857 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda64904-eb0f-4fbc-aeb9-afe296519d0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:39.736495 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda64904-eb0f-4fbc-aeb9-afe296519d0e None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:39.736768 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda64904-eb0f-4fbc-aeb9-afe296519d0e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:39.737035 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda64904-eb0f-4fbc-aeb9-afe296519d0e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:39.801135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda64904-eb0f-4fbc-aeb9-afe296519d0e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:39.802553 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda64904-eb0f-4fbc-aeb9-afe296519d0e None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:39.875781 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:39.896463 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 575/1144] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:06:39 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:06:39.902930 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab75179-2f82-40ef-8697-ab2dae64d579 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:39.982546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:40.001219 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab75179-2f82-40ef-8697-ab2dae64d579 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:40.035731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe7fff4-6b9b-493c-b5c4-9f985962ffaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:40.036366 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe7fff4-6b9b-493c-b5c4-9f985962ffaf None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:40.036645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe7fff4-6b9b-493c-b5c4-9f985962ffaf None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:40.036913 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe7fff4-6b9b-493c-b5c4-9f985962ffaf None admin] PATH_INFO: `/v3/projects/d1e29ad1cecd41ab82df6a576edf6729` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:40.038015 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe7fff4-6b9b-493c-b5c4-9f985962ffaf None admin] RBAC: Authorizing `identity:get_project(project_id=d1e29ad1cecd41ab82df6a576edf6729)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:40.039425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe7fff4-6b9b-493c-b5c4-9f985962ffaf None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:40.041082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 570/1145] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:06:39 2022] GET /identity/v3/projects/d1e29ad1cecd41ab82df6a576edf6729 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:40.047114 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda64904-eb0f-4fbc-aeb9-afe296519d0e None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:40.141755 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda64904-eb0f-4fbc-aeb9-afe296519d0e None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:40.174384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f2b2ab-a401-4022-8380-6103ceec874d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:40.174971 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f2b2ab-a401-4022-8380-6103ceec874d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:40.175240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f2b2ab-a401-4022-8380-6103ceec874d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:40.175502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f2b2ab-a401-4022-8380-6103ceec874d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:40.176237 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f2b2ab-a401-4022-8380-6103ceec874d None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=d1e29ad1cecd41ab82df6a576edf6729, user.id=d72058ab2d614b3f96e68e1428cfd45e) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:40.176537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f2b2ab-a401-4022-8380-6103ceec874d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:40.177920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f2b2ab-a401-4022-8380-6103ceec874d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:40.183087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 576/1146] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:06:40 2022] GET /identity/v3/role_assignments?user.id=d72058ab2d614b3f96e68e1428cfd45e&scope.project.id=d1e29ad1cecd41ab82df6a576edf6729&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 259 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:42.021469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad23a5fc-5a57-404b-b982-e9e2225f5b14 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:42.022026 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad23a5fc-5a57-404b-b982-e9e2225f5b14 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:42.022026 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad23a5fc-5a57-404b-b982-e9e2225f5b14 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:42.023335 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 571/1147] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:42.028571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c45084-1d44-40b8-92d9-90a65345ea06 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:42.029014 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c45084-1d44-40b8-92d9-90a65345ea06 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:42.029128 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c45084-1d44-40b8-92d9-90a65345ea06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:42.046472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-58c45084-1d44-40b8-92d9-90a65345ea06 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:42.047630 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-58c45084-1d44-40b8-92d9-90a65345ea06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:42.081833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:42.111835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 577/1148] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:42.378763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad23a5fc-5a57-404b-b982-e9e2225f5b14 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:42.476384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad23a5fc-5a57-404b-b982-e9e2225f5b14 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:42.509185 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16de6777-20c5-44d9-b983-643ca2fbf0cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:42.509790 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16de6777-20c5-44d9-b983-643ca2fbf0cc None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:42.510061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16de6777-20c5-44d9-b983-643ca2fbf0cc None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:42.510325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16de6777-20c5-44d9-b983-643ca2fbf0cc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:42.520029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16de6777-20c5-44d9-b983-643ca2fbf0cc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:06:42.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-16de6777-20c5-44d9-b983-643ca2fbf0cc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:06:42.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:42.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:42.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:42.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:42.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:42.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:42.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:42.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:42.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:42.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:42.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:42.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:06:42.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:06:42.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:06:42.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:06:42.525767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:06:42.525767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:06:42.525767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:06:42.525767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:06:42.525767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:42.525767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:42.525767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:42.525767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:42.525767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:42.525767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:42.525767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:42.525767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:42.525767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:42.525767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:42.525767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:42.525767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:42.526855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:42.526855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:42.526855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:42.526855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:42.526855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:06:42.526855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:06:42.526855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:06:42.526855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:06:42.526855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:06:42.526855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:06:42.526855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:06:42.526855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:42.526855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 572/1149] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:06:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:42.531585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c45084-1d44-40b8-92d9-90a65345ea06 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:42.626294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c45084-1d44-40b8-92d9-90a65345ea06 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:42.658509 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181742f0-8175-4433-bc1b-a32fefe0ba71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:42.659095 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181742f0-8175-4433-bc1b-a32fefe0ba71 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:42.659370 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181742f0-8175-4433-bc1b-a32fefe0ba71 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:42.659634 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181742f0-8175-4433-bc1b-a32fefe0ba71 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:42.660293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181742f0-8175-4433-bc1b-a32fefe0ba71 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:42.660589 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181742f0-8175-4433-bc1b-a32fefe0ba71 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:42.661955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181742f0-8175-4433-bc1b-a32fefe0ba71 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:42.670186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 578/1150] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:06:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:42.675791 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16de6777-20c5-44d9-b983-643ca2fbf0cc None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:42.687107 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:42.776928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16de6777-20c5-44d9-b983-643ca2fbf0cc None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:42.809794 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe344bb-251a-4386-a29e-da7cd053d618 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:42.810394 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe344bb-251a-4386-a29e-da7cd053d618 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:42.810668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe344bb-251a-4386-a29e-da7cd053d618 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:42.810934 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe344bb-251a-4386-a29e-da7cd053d618 None admin] PATH_INFO: `/v3/users/d72058ab2d614b3f96e68e1428cfd45e` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:42.812035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe344bb-251a-4386-a29e-da7cd053d618 None admin] RBAC: Authorizing `identity:get_user(user_id=d72058ab2d614b3f96e68e1428cfd45e)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:42.813739 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe344bb-251a-4386-a29e-da7cd053d618 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:42.815306 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 573/1151] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:06:42 2022] GET /identity/v3/users/d72058ab2d614b3f96e68e1428cfd45e => generated 286 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:42.821015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181742f0-8175-4433-bc1b-a32fefe0ba71 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:42.854247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:42.917848 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181742f0-8175-4433-bc1b-a32fefe0ba71 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:42.949679 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ca0581-9165-423b-9c4f-e1ba59edcc51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:42.950289 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ca0581-9165-423b-9c4f-e1ba59edcc51 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:42.950564 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ca0581-9165-423b-9c4f-e1ba59edcc51 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:42.950829 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ca0581-9165-423b-9c4f-e1ba59edcc51 None admin] PATH_INFO: `/v3/projects/d1e29ad1cecd41ab82df6a576edf6729` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:42.951911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ca0581-9165-423b-9c4f-e1ba59edcc51 None admin] RBAC: Authorizing `identity:get_project(project_id=d1e29ad1cecd41ab82df6a576edf6729)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:42.953356 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ca0581-9165-423b-9c4f-e1ba59edcc51 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:42.954964 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 579/1152] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:06:42 2022] GET /identity/v3/projects/d1e29ad1cecd41ab82df6a576edf6729 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:42.960886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe344bb-251a-4386-a29e-da7cd053d618 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:43.036076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:43.057699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe344bb-251a-4386-a29e-da7cd053d618 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:43.091687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96595460-069d-4c22-8e88-67a7318db862 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:43.092344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96595460-069d-4c22-8e88-67a7318db862 None admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:43.092617 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96595460-069d-4c22-8e88-67a7318db862 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:43.092886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96595460-069d-4c22-8e88-67a7318db862 None admin] PATH_INFO: `/v3/projects/d1e29ad1cecd41ab82df6a576edf6729/users/d72058ab2d614b3f96e68e1428cfd45e/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:43.094763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96595460-069d-4c22-8e88-67a7318db862 None admin] RBAC: Authorizing `identity:create_grant(project_id=d1e29ad1cecd41ab82df6a576edf6729, user_id=d72058ab2d614b3f96e68e1428cfd45e, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:43.096197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96595460-069d-4c22-8e88-67a7318db862 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:43.178891 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 574/1153] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:06:42 2022] PUT /identity/v3/projects/d1e29ad1cecd41ab82df6a576edf6729/users/d72058ab2d614b3f96e68e1428cfd45e/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:06:45.037107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db339981-f199-43b6-b8a2-471e8af0b899 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:45.037653 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db339981-f199-43b6-b8a2-471e8af0b899 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:45.037653 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db339981-f199-43b6-b8a2-471e8af0b899 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:45.038983 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 580/1154] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:45.044245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10af7a7f-050b-4af6-a435-1ca4d01e8064 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:45.044656 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10af7a7f-050b-4af6-a435-1ca4d01e8064 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:45.044763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10af7a7f-050b-4af6-a435-1ca4d01e8064 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:45.063647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-10af7a7f-050b-4af6-a435-1ca4d01e8064 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:45.064847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-10af7a7f-050b-4af6-a435-1ca4d01e8064 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:45.130947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 575/1155] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:45.400915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db339981-f199-43b6-b8a2-471e8af0b899 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:45.500001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db339981-f199-43b6-b8a2-471e8af0b899 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:45.501528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:45.532478 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5683b293-4de7-43e0-8b55-a3346fb37b40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:45.533106 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5683b293-4de7-43e0-8b55-a3346fb37b40 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:45.533557 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5683b293-4de7-43e0-8b55-a3346fb37b40 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:45.533839 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5683b293-4de7-43e0-8b55-a3346fb37b40 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:45.544563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5683b293-4de7-43e0-8b55-a3346fb37b40 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:06:45.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-5683b293-4de7-43e0-8b55-a3346fb37b40 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:06:45.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:45.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:45.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:45.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:45.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:45.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:45.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:45.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:45.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:45.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:45.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:45.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:06:45.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:06:45.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:06:45.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:06:45.550308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:06:45.550308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:06:45.550308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:06:45.550308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:06:45.550308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:45.550308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:45.550308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:45.550308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:45.550308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:45.550308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:45.550308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:45.550308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:45.550308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:45.550308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:45.550308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:45.550308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:45.551440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:45.551440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:45.551440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:45.551440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:45.551440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:06:45.551440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:06:45.551440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:06:45.551440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:06:45.551440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:06:45.551440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:06:45.551440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:06:45.551440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:45.551440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 581/1156] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:06:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:45.556425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10af7a7f-050b-4af6-a435-1ca4d01e8064 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:45.619526 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:45.653640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10af7a7f-050b-4af6-a435-1ca4d01e8064 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:45.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d913d3-6857-4761-809f-4ce201629e92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:45.687815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d913d3-6857-4761-809f-4ce201629e92 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:45.688084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d913d3-6857-4761-809f-4ce201629e92 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:45.688384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d913d3-6857-4761-809f-4ce201629e92 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:45.689033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d913d3-6857-4761-809f-4ce201629e92 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:45.689331 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d913d3-6857-4761-809f-4ce201629e92 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:45.690674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d913d3-6857-4761-809f-4ce201629e92 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:45.699053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 576/1157] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:06:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:45.704750 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5683b293-4de7-43e0-8b55-a3346fb37b40 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:45.802945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5683b293-4de7-43e0-8b55-a3346fb37b40 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:45.836320 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a372ed9-45b1-4bcf-8739-4b9d0a8d4c43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:45.836933 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a372ed9-45b1-4bcf-8739-4b9d0a8d4c43 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:45.837206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a372ed9-45b1-4bcf-8739-4b9d0a8d4c43 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:45.837469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a372ed9-45b1-4bcf-8739-4b9d0a8d4c43 None admin] PATH_INFO: `/v3/users/d72058ab2d614b3f96e68e1428cfd45e` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:45.838558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a372ed9-45b1-4bcf-8739-4b9d0a8d4c43 None admin] RBAC: Authorizing `identity:get_user(user_id=d72058ab2d614b3f96e68e1428cfd45e)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:45.840526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a372ed9-45b1-4bcf-8739-4b9d0a8d4c43 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:45.842090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 582/1158] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:06:45 2022] GET /identity/v3/users/d72058ab2d614b3f96e68e1428cfd45e => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:45.848002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d913d3-6857-4761-809f-4ce201629e92 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:45.945469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d913d3-6857-4761-809f-4ce201629e92 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:45.970307 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:45.977797 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da02fca5-b044-496e-a1e2-10d90ca0bdab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:45.983266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da02fca5-b044-496e-a1e2-10d90ca0bdab None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:45.983266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da02fca5-b044-496e-a1e2-10d90ca0bdab None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:45.983266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da02fca5-b044-496e-a1e2-10d90ca0bdab None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:46.045100 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da02fca5-b044-496e-a1e2-10d90ca0bdab None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:46.046536 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da02fca5-b044-496e-a1e2-10d90ca0bdab None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:46.140952 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 577/1159] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:06:45 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 21 01:06:46.146841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a372ed9-45b1-4bcf-8739-4b9d0a8d4c43 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:46.178802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:46.246467 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a372ed9-45b1-4bcf-8739-4b9d0a8d4c43 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:46.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab64b48-beb5-4588-b423-8ad3f720ba4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:46.280366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab64b48-beb5-4588-b423-8ad3f720ba4c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:46.280643 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab64b48-beb5-4588-b423-8ad3f720ba4c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:46.280912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab64b48-beb5-4588-b423-8ad3f720ba4c None admin] PATH_INFO: `/v3/projects/d1e29ad1cecd41ab82df6a576edf6729` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:46.282021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab64b48-beb5-4588-b423-8ad3f720ba4c None admin] RBAC: Authorizing `identity:get_project(project_id=d1e29ad1cecd41ab82df6a576edf6729)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:46.283436 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab64b48-beb5-4588-b423-8ad3f720ba4c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:46.285104 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 583/1160] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:06:46 2022] GET /identity/v3/projects/d1e29ad1cecd41ab82df6a576edf6729 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:46.291299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da02fca5-b044-496e-a1e2-10d90ca0bdab None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:46.317845 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:46.390682 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da02fca5-b044-496e-a1e2-10d90ca0bdab None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:46.423709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d55c31-153b-4cb5-bcfe-b3241b7fb32a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:46.424328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d55c31-153b-4cb5-bcfe-b3241b7fb32a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:46.424600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d55c31-153b-4cb5-bcfe-b3241b7fb32a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:46.424864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d55c31-153b-4cb5-bcfe-b3241b7fb32a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:46.425567 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d55c31-153b-4cb5-bcfe-b3241b7fb32a None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=d1e29ad1cecd41ab82df6a576edf6729, user.id=d72058ab2d614b3f96e68e1428cfd45e) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:46.425866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d55c31-153b-4cb5-bcfe-b3241b7fb32a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:46.427264 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d55c31-153b-4cb5-bcfe-b3241b7fb32a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:46.433480 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 578/1161] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:06:46 2022] GET /identity/v3/role_assignments?user.id=d72058ab2d614b3f96e68e1428cfd45e&scope.project.id=d1e29ad1cecd41ab82df6a576edf6729&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:48.358788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bc6d1c-781e-4b56-86fa-fcf6d315cdaa None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:48.359178 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bc6d1c-781e-4b56-86fa-fcf6d315cdaa None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:48.359332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bc6d1c-781e-4b56-86fa-fcf6d315cdaa None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:48.360724 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 584/1162] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:48.366038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64c046a-a80d-4ca9-b4a6-f784ebaedc12 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:48.366460 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64c046a-a80d-4ca9-b4a6-f784ebaedc12 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:48.366576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64c046a-a80d-4ca9-b4a6-f784ebaedc12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:48.383999 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-d64c046a-a80d-4ca9-b4a6-f784ebaedc12 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:48.385688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-d64c046a-a80d-4ca9-b4a6-f784ebaedc12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:48.450664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 579/1163] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:48.716376 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bc6d1c-781e-4b56-86fa-fcf6d315cdaa None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:48.809913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:48.814642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bc6d1c-781e-4b56-86fa-fcf6d315cdaa None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:48.846827 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4174537-81fd-4c71-a182-010a4c3ca7cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:48.847418 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4174537-81fd-4c71-a182-010a4c3ca7cb None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:48.847693 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4174537-81fd-4c71-a182-010a4c3ca7cb None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:48.847962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4174537-81fd-4c71-a182-010a4c3ca7cb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:48.913567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4174537-81fd-4c71-a182-010a4c3ca7cb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:48.914984 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4174537-81fd-4c71-a182-010a4c3ca7cb None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:49.009142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:49.013971 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 585/1164] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:06:48 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:06:49.020246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64c046a-a80d-4ca9-b4a6-f784ebaedc12 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:49.043833 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:49.117037 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64c046a-a80d-4ca9-b4a6-f784ebaedc12 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:49.149849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62b1b7d-7615-48f4-b7f9-a2e30a947cd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:49.150462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62b1b7d-7615-48f4-b7f9-a2e30a947cd7 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:49.150764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62b1b7d-7615-48f4-b7f9-a2e30a947cd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:49.151031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62b1b7d-7615-48f4-b7f9-a2e30a947cd7 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:49.157629 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62b1b7d-7615-48f4-b7f9-a2e30a947cd7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:49.159043 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62b1b7d-7615-48f4-b7f9-a2e30a947cd7 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-e62b1b7d-7615-48f4-b7f9-a2e30a947cd7 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:49.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:49.171831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:49.171831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:49.171831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:49.171831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:49.171831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:49.171831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:49.171831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:06:49.171831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:06:49.171831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:06:49.171831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:06:49.171831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:49.171831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:49.171831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:49.171831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:49.171831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:49.171831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:49.173224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:49.173224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:49.173224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:49.173224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:49.173224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:49.173224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:49.173224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:49.173224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:49.173224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:49.173224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:49.173224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Aug 21 01:06:49.173224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:06:49.173224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 21 01:06:49.173224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:06:49.173224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 580/1165] 10.222.0.47 () {56 vars in 1117 bytes} [Sun Aug 21 01:06:49 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:49.177820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4174537-81fd-4c71-a182-010a4c3ca7cb None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:49.273689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4174537-81fd-4c71-a182-010a4c3ca7cb None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:49.306812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e610d1-0226-4e09-8d0b-ea8e532c12d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:49.308003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e610d1-0226-4e09-8d0b-ea8e532c12d3 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:49.308298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e610d1-0226-4e09-8d0b-ea8e532c12d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:49.308564 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e610d1-0226-4e09-8d0b-ea8e532c12d3 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:49.309180 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e610d1-0226-4e09-8d0b-ea8e532c12d3 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:49.309472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e610d1-0226-4e09-8d0b-ea8e532c12d3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:49.310797 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e610d1-0226-4e09-8d0b-ea8e532c12d3 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:49.317816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 586/1166] 10.222.0.47 () {56 vars in 1115 bytes} [Sun Aug 21 01:06:49 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:51.179116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4aac3b3-5186-4669-a3a6-b33a4d0160a9 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:51.179647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4aac3b3-5186-4669-a3a6-b33a4d0160a9 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:51.179647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4aac3b3-5186-4669-a3a6-b33a4d0160a9 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:51.181022 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 581/1167] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:51.187184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d6f5c0-c1bd-4587-8394-e72b59819bf3 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:51.187601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d6f5c0-c1bd-4587-8394-e72b59819bf3 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:51.187721 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d6f5c0-c1bd-4587-8394-e72b59819bf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:51.205984 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-56d6f5c0-c1bd-4587-8394-e72b59819bf3 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:51.207169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-56d6f5c0-c1bd-4587-8394-e72b59819bf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:51.261688 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:51.271803 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 587/1168] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:51.552671 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4aac3b3-5186-4669-a3a6-b33a4d0160a9 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:51.637125 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:51.656354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4aac3b3-5186-4669-a3a6-b33a4d0160a9 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:51.690532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be960f7-b805-4cb5-ab04-5a37a88512c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:51.691134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be960f7-b805-4cb5-ab04-5a37a88512c2 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:51.691412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be960f7-b805-4cb5-ab04-5a37a88512c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:51.691685 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be960f7-b805-4cb5-ab04-5a37a88512c2 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:51.692372 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be960f7-b805-4cb5-ab04-5a37a88512c2 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:51.693733 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be960f7-b805-4cb5-ab04-5a37a88512c2 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:51.762828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 582/1169] 10.222.0.47 () {60 vars in 1138 bytes} [Sun Aug 21 01:06:51 2022] POST /identity/v3/domains => generated 267 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:06:53.679414 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f170acd-f0e6-444f-8faa-4e5bffb5a322 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:53.679959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f170acd-f0e6-444f-8faa-4e5bffb5a322 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:53.679959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f170acd-f0e6-444f-8faa-4e5bffb5a322 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:53.681230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 588/1170] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:53.687546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f81cca1-8dc2-4373-aa77-221b880b7611 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:53.688040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f81cca1-8dc2-4373-aa77-221b880b7611 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:53.688166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f81cca1-8dc2-4373-aa77-221b880b7611 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:53.706102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-2f81cca1-8dc2-4373-aa77-221b880b7611 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:53.707300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-2f81cca1-8dc2-4373-aa77-221b880b7611 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:53.775118 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 583/1171] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:54.045067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f170acd-f0e6-444f-8faa-4e5bffb5a322 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:54.145281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f170acd-f0e6-444f-8faa-4e5bffb5a322 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:54.180536 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b50403-64c1-4807-aaf8-fabc2c39331f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:54.181158 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b50403-64c1-4807-aaf8-fabc2c39331f None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:54.181458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b50403-64c1-4807-aaf8-fabc2c39331f None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:54.182216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b50403-64c1-4807-aaf8-fabc2c39331f None admin] PATH_INFO: `/v3/domains/5c18b73c395042038904df1a351ca9c8` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:54.192541 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b50403-64c1-4807-aaf8-fabc2c39331f None admin] RBAC: Authorizing `identity:get_domain(domain_id=5c18b73c395042038904df1a351ca9c8)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:54.193974 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b50403-64c1-4807-aaf8-fabc2c39331f None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:54.195572 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 589/1172] 10.222.0.47 () {56 vars in 1183 bytes} [Sun Aug 21 01:06:54 2022] GET /identity/v3/domains/5c18b73c395042038904df1a351ca9c8 => generated 267 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:54.202229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f81cca1-8dc2-4373-aa77-221b880b7611 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:54.269501 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:54.297897 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f81cca1-8dc2-4373-aa77-221b880b7611 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:54.331260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8124c540-bf7c-44a8-92f9-631472b427c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:54.331854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8124c540-bf7c-44a8-92f9-631472b427c3 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:54.332159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8124c540-bf7c-44a8-92f9-631472b427c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:54.332429 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8124c540-bf7c-44a8-92f9-631472b427c3 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:54.333073 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8124c540-bf7c-44a8-92f9-631472b427c3 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:54.334493 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8124c540-bf7c-44a8-92f9-631472b427c3 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:54.481877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 584/1173] 10.222.0.47 () {60 vars in 1141 bytes} [Sun Aug 21 01:06:54 2022] POST /identity/v3/projects => generated 361 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:06:56.383006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaea972b-4461-4fce-97ee-cf38359135d5 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:56.383469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaea972b-4461-4fce-97ee-cf38359135d5 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:56.383583 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaea972b-4461-4fce-97ee-cf38359135d5 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:56.384936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 590/1174] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:56.390082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ec2b35-85ae-423a-b664-980c9530d435 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:56.390476 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ec2b35-85ae-423a-b664-980c9530d435 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:56.390630 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ec2b35-85ae-423a-b664-980c9530d435 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:56.408821 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-07ec2b35-85ae-423a-b664-980c9530d435 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:56.410011 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-07ec2b35-85ae-423a-b664-980c9530d435 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:56.476557 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 585/1175] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:06:56.742902 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaea972b-4461-4fce-97ee-cf38359135d5 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:56.795413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:56.842391 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaea972b-4461-4fce-97ee-cf38359135d5 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:56.874206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718b6777-2d2a-4a8e-beb3-fc230082bb3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:56.874816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718b6777-2d2a-4a8e-beb3-fc230082bb3a None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:56.875090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718b6777-2d2a-4a8e-beb3-fc230082bb3a None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:56.875358 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718b6777-2d2a-4a8e-beb3-fc230082bb3a None admin] PATH_INFO: `/v3/domains/5c18b73c395042038904df1a351ca9c8` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:56.876466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718b6777-2d2a-4a8e-beb3-fc230082bb3a None admin] RBAC: Authorizing `identity:get_domain(domain_id=5c18b73c395042038904df1a351ca9c8)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:56.877860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718b6777-2d2a-4a8e-beb3-fc230082bb3a None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:56.879400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 591/1176] 10.222.0.47 () {56 vars in 1183 bytes} [Sun Aug 21 01:06:56 2022] GET /identity/v3/domains/5c18b73c395042038904df1a351ca9c8 => generated 267 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:56.885609 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ec2b35-85ae-423a-b664-980c9530d435 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:56.938755 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:56.982079 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ec2b35-85ae-423a-b664-980c9530d435 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:57.016175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e795bc8-dfce-4cd7-b22c-e8509dd3db55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:57.016702 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e795bc8-dfce-4cd7-b22c-e8509dd3db55 None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:57.016977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e795bc8-dfce-4cd7-b22c-e8509dd3db55 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:57.017247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e795bc8-dfce-4cd7-b22c-e8509dd3db55 None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:57.017888 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e795bc8-dfce-4cd7-b22c-e8509dd3db55 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:57.019305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e795bc8-dfce-4cd7-b22c-e8509dd3db55 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:57.123542 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 586/1177] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:06:56 2022] POST /identity/v3/users => generated 311 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:06:59.034417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e685351-5ff9-4a00-a413-a08780a89f7f None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:59.034992 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e685351-5ff9-4a00-a413-a08780a89f7f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:59.034992 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e685351-5ff9-4a00-a413-a08780a89f7f None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:59.036321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 592/1178] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:06:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:06:59.041741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c037052d-b4e2-4740-9dde-6f6828f82706 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:59.042156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c037052d-b4e2-4740-9dde-6f6828f82706 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:59.042284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c037052d-b4e2-4740-9dde-6f6828f82706 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:59.059800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-c037052d-b4e2-4740-9dde-6f6828f82706 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:06:59.061013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c037052d-b4e2-4740-9dde-6f6828f82706 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:06:59.126468 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 587/1179] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:06:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:06:59.397729 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e685351-5ff9-4a00-a413-a08780a89f7f None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:59.495903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e685351-5ff9-4a00-a413-a08780a89f7f None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:59.515141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:59.528186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d51c79-9549-484a-b4f5-31cc756a0115 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:59.528804 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d51c79-9549-484a-b4f5-31cc756a0115 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:59.529076 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d51c79-9549-484a-b4f5-31cc756a0115 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:59.529341 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d51c79-9549-484a-b4f5-31cc756a0115 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:59.539116 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d51c79-9549-484a-b4f5-31cc756a0115 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:06:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-71d51c79-9549-484a-b4f5-31cc756a0115 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:06:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:06:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:06:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:06:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:06:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:06:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:06:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:06:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:06:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:06:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:06:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:06:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:06:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:06:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:06:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:06:59.545502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:06:59.545502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:06:59.545502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:06:59.545502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:06:59.545502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:06:59.545502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:06:59.545502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:06:59.545502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:06:59.545502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:06:59.545502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:06:59.545502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:06:59.545502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:06:59.545502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:06:59.545502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:06:59.545502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:06:59.545502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:06:59.546772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:06:59.546772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:06:59.546772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:06:59.546772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:06:59.546772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:06:59.546772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:06:59.546772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:06:59.546772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:06:59.546772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:06:59.546772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:06:59.546772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:06:59.546772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:06:59.546772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 593/1180] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:06:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:06:59.551597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c037052d-b4e2-4740-9dde-6f6828f82706 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:59.592170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:59.651218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c037052d-b4e2-4740-9dde-6f6828f82706 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:59.684765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172afde0-7ca8-4195-879f-370db699e3fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:59.685355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172afde0-7ca8-4195-879f-370db699e3fd None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:59.685624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172afde0-7ca8-4195-879f-370db699e3fd None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:59.685889 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172afde0-7ca8-4195-879f-370db699e3fd None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:59.686797 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172afde0-7ca8-4195-879f-370db699e3fd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:06:59.687089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172afde0-7ca8-4195-879f-370db699e3fd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:59.688596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172afde0-7ca8-4195-879f-370db699e3fd None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:59.696344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 588/1181] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:06:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:59.702154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d51c79-9549-484a-b4f5-31cc756a0115 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:59.797366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d51c79-9549-484a-b4f5-31cc756a0115 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:59.831228 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f42747-e9cb-44c0-b1b4-7353e0291eae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:59.831831 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f42747-e9cb-44c0-b1b4-7353e0291eae None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:59.832129 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f42747-e9cb-44c0-b1b4-7353e0291eae None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:59.832393 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f42747-e9cb-44c0-b1b4-7353e0291eae None admin] PATH_INFO: `/v3/users/7971d60f7133455ba9504422b8e86438` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:06:59.844895 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f42747-e9cb-44c0-b1b4-7353e0291eae None admin] RBAC: Authorizing `identity:get_user(user_id=7971d60f7133455ba9504422b8e86438)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:06:59.846318 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f42747-e9cb-44c0-b1b4-7353e0291eae None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:06:59.847875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 594/1182] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:06:59 2022] GET /identity/v3/users/7971d60f7133455ba9504422b8e86438 => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:06:59.854334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172afde0-7ca8-4195-879f-370db699e3fd None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:06:59.951388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172afde0-7ca8-4195-879f-370db699e3fd None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:06:59.952827 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:06:59.984280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733deb3b-7cd2-41b3-835a-f146ccb9f1e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:06:59.984974 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733deb3b-7cd2-41b3-835a-f146ccb9f1e4 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:06:59.985246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733deb3b-7cd2-41b3-835a-f146ccb9f1e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:06:59.985512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733deb3b-7cd2-41b3-835a-f146ccb9f1e4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:00.050874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733deb3b-7cd2-41b3-835a-f146ccb9f1e4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:07:00.052329 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733deb3b-7cd2-41b3-835a-f146ccb9f1e4 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:07:00.145249 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 589/1183] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:06:59 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:07:00.151641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f42747-e9cb-44c0-b1b4-7353e0291eae None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:07:00.191682 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:07:00.250095 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f42747-e9cb-44c0-b1b4-7353e0291eae None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:07:00.283546 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292b01ab-4efe-4f4b-beef-512463e4fa9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:07:00.284195 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292b01ab-4efe-4f4b-beef-512463e4fa9a None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:00.284479 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292b01ab-4efe-4f4b-beef-512463e4fa9a None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:00.284756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292b01ab-4efe-4f4b-beef-512463e4fa9a None admin] PATH_INFO: `/v3/projects/d77f6264687e403e8019bc11cc17aefe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:00.285870 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292b01ab-4efe-4f4b-beef-512463e4fa9a None admin] RBAC: Authorizing `identity:get_project(project_id=d77f6264687e403e8019bc11cc17aefe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:07:00.287286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292b01ab-4efe-4f4b-beef-512463e4fa9a None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:07:00.288956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 595/1184] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:07:00 2022] GET /identity/v3/projects/d77f6264687e403e8019bc11cc17aefe => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:07:00.295541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733deb3b-7cd2-41b3-835a-f146ccb9f1e4 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:07:00.297753 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:07:00.394758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733deb3b-7cd2-41b3-835a-f146ccb9f1e4 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:07:00.427825 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3583f261-21f1-4d22-a2ea-d996ec4d72a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:07:00.428709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3583f261-21f1-4d22-a2ea-d996ec4d72a0 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:00.429026 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3583f261-21f1-4d22-a2ea-d996ec4d72a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:00.429363 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3583f261-21f1-4d22-a2ea-d996ec4d72a0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:00.430085 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3583f261-21f1-4d22-a2ea-d996ec4d72a0 None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=d77f6264687e403e8019bc11cc17aefe, user.id=7971d60f7133455ba9504422b8e86438) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:07:00.430392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3583f261-21f1-4d22-a2ea-d996ec4d72a0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:07:00.432519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3583f261-21f1-4d22-a2ea-d996ec4d72a0 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:07:00.438115 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 590/1185] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:07:00 2022] GET /identity/v3/role_assignments?user.id=7971d60f7133455ba9504422b8e86438&scope.project.id=d77f6264687e403e8019bc11cc17aefe&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:07:02.298215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3f00d2-47ec-438f-9d71-34703285090f None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:02.298774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3f00d2-47ec-438f-9d71-34703285090f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:02.298774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3f00d2-47ec-438f-9d71-34703285090f None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:02.300080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 596/1186] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:07:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:07:02.305316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ef702f-a0ff-453a-bb3d-ea8d12b48f88 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:02.305770 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ef702f-a0ff-453a-bb3d-ea8d12b48f88 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:02.305871 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ef702f-a0ff-453a-bb3d-ea8d12b48f88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:02.323793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-c4ef702f-a0ff-453a-bb3d-ea8d12b48f88 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:07:02.325037 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c4ef702f-a0ff-453a-bb3d-ea8d12b48f88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:07:02.384049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:07:02.391206 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 591/1187] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:07:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:07:02.665675 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3f00d2-47ec-438f-9d71-34703285090f None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:07:02.764363 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3f00d2-47ec-438f-9d71-34703285090f None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:07:02.771431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:07:02.797655 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3909ee-564c-4efd-9c89-233d713a7cbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:07:02.798270 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3909ee-564c-4efd-9c89-233d713a7cbd None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:02.798542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3909ee-564c-4efd-9c89-233d713a7cbd None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:02.798806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3909ee-564c-4efd-9c89-233d713a7cbd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:02.808538 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3909ee-564c-4efd-9c89-233d713a7cbd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:07:02.813398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-9d3909ee-564c-4efd-9c89-233d713a7cbd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:07:02.813398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:07:02.813398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:07:02.813398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:07:02.813398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:07:02.813398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:07:02.813398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:07:02.813398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:07:02.813398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:07:02.813398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:07:02.813398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:07:02.813398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:07:02.813398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:07:02.813398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:07:02.813398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:07:02.813398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:07:02.814526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:07:02.814526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:07:02.814526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:07:02.814526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:07:02.814526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:07:02.814526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:07:02.814526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:07:02.814526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:07:02.814526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:07:02.814526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:07:02.814526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:07:02.814526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:07:02.814526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:07:02.814526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:07:02.814526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:07:02.814526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:07:02.815777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:07:02.815777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:07:02.815777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:07:02.815777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:07:02.815777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:07:02.815777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:07:02.815777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:07:02.815777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:07:02.815777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:07:02.815777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:07:02.815777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:07:02.815777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:07:02.815777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 597/1188] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:07:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:07:02.820558 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ef702f-a0ff-453a-bb3d-ea8d12b48f88 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:07:02.917488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ef702f-a0ff-453a-bb3d-ea8d12b48f88 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:07:02.950879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0932a12-f133-445e-a808-1cdfa6f3cf35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:07:02.951472 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0932a12-f133-445e-a808-1cdfa6f3cf35 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:02.951742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0932a12-f133-445e-a808-1cdfa6f3cf35 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:02.952006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0932a12-f133-445e-a808-1cdfa6f3cf35 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:02.952685 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0932a12-f133-445e-a808-1cdfa6f3cf35 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:07:02.952978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0932a12-f133-445e-a808-1cdfa6f3cf35 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:07:02.954317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0932a12-f133-445e-a808-1cdfa6f3cf35 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:07:02.962046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 592/1189] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:07:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:07:02.967984 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3909ee-564c-4efd-9c89-233d713a7cbd None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:07:03.065152 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3909ee-564c-4efd-9c89-233d713a7cbd None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:07:03.098910 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1c5ee4-9c9a-41fc-9120-cd936a1dddbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:07:03.099536 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1c5ee4-9c9a-41fc-9120-cd936a1dddbd None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:03.099815 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1c5ee4-9c9a-41fc-9120-cd936a1dddbd None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:03.100085 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1c5ee4-9c9a-41fc-9120-cd936a1dddbd None admin] PATH_INFO: `/v3/users/7971d60f7133455ba9504422b8e86438` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:03.101231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1c5ee4-9c9a-41fc-9120-cd936a1dddbd None admin] RBAC: Authorizing `identity:get_user(user_id=7971d60f7133455ba9504422b8e86438)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:07:03.102673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1c5ee4-9c9a-41fc-9120-cd936a1dddbd None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:07:03.104284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 598/1190] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:07:02 2022] GET /identity/v3/users/7971d60f7133455ba9504422b8e86438 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:07:03.110160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0932a12-f133-445e-a808-1cdfa6f3cf35 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:07:03.168475 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:07:03.209521 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0932a12-f133-445e-a808-1cdfa6f3cf35 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:07:03.241521 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e8584d-6a90-4a61-9e82-3a5e2407fcfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:07:03.242138 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e8584d-6a90-4a61-9e82-3a5e2407fcfc None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:03.242414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e8584d-6a90-4a61-9e82-3a5e2407fcfc None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:03.242684 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e8584d-6a90-4a61-9e82-3a5e2407fcfc None admin] PATH_INFO: `/v3/projects/d77f6264687e403e8019bc11cc17aefe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:03.244339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e8584d-6a90-4a61-9e82-3a5e2407fcfc None admin] RBAC: Authorizing `identity:get_project(project_id=d77f6264687e403e8019bc11cc17aefe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:07:03.245753 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e8584d-6a90-4a61-9e82-3a5e2407fcfc None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:07:03.247370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 593/1191] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:07:03 2022] GET /identity/v3/projects/d77f6264687e403e8019bc11cc17aefe => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:07:03.253265 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1c5ee4-9c9a-41fc-9120-cd936a1dddbd None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:07:03.287041 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:07:03.350890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1c5ee4-9c9a-41fc-9120-cd936a1dddbd None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:07:03.384685 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db2bbbc-1873-4065-9a92-1c0aebc7990e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:07:03.385334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db2bbbc-1873-4065-9a92-1c0aebc7990e None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:03.385612 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db2bbbc-1873-4065-9a92-1c0aebc7990e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:03.385883 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db2bbbc-1873-4065-9a92-1c0aebc7990e None admin] PATH_INFO: `/v3/projects/d77f6264687e403e8019bc11cc17aefe/users/7971d60f7133455ba9504422b8e86438/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:03.387755 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db2bbbc-1873-4065-9a92-1c0aebc7990e None admin] RBAC: Authorizing `identity:create_grant(project_id=d77f6264687e403e8019bc11cc17aefe, user_id=7971d60f7133455ba9504422b8e86438, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:07:03.389207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db2bbbc-1873-4065-9a92-1c0aebc7990e None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:07:03.456932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 599/1192] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:07:03 2022] PUT /identity/v3/projects/d77f6264687e403e8019bc11cc17aefe/users/7971d60f7133455ba9504422b8e86438/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:07:05.355618 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc5695d-dd22-49c5-86bc-64a39ebfb890 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:05.356064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc5695d-dd22-49c5-86bc-64a39ebfb890 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:05.356177 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc5695d-dd22-49c5-86bc-64a39ebfb890 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:05.357521 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 594/1193] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:07:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:07:05.362826 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9d134-d62e-400b-9356-cef71bac1b83 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:05.363270 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9d134-d62e-400b-9356-cef71bac1b83 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:05.363403 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9d134-d62e-400b-9356-cef71bac1b83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:05.388470 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-afd9d134-d62e-400b-9356-cef71bac1b83 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:07:05.389740 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-afd9d134-d62e-400b-9356-cef71bac1b83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:07:05.463192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 600/1194] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:07:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:07:05.736388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc5695d-dd22-49c5-86bc-64a39ebfb890 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:07:05.840380 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc5695d-dd22-49c5-86bc-64a39ebfb890 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:07:05.866682 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:07:05.875705 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d863eb2e-9e6e-408b-9923-c31a9a2fdadb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:07:05.877637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d863eb2e-9e6e-408b-9923-c31a9a2fdadb None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:05.877637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d863eb2e-9e6e-408b-9923-c31a9a2fdadb None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:05.877637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d863eb2e-9e6e-408b-9923-c31a9a2fdadb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:05.887797 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d863eb2e-9e6e-408b-9923-c31a9a2fdadb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:07:05.892417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-d863eb2e-9e6e-408b-9923-c31a9a2fdadb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:07:05.892417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:07:05.892417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:07:05.892417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:07:05.892417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:07:05.892417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:07:05.892417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:07:05.892417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:07:05.892417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:07:05.892417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:07:05.892417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:07:05.892417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:07:05.892417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:07:05.892417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:07:05.892417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:07:05.892417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:07:05.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:07:05.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:07:05.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:07:05.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:07:05.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:07:05.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:07:05.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:07:05.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:07:05.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:07:05.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:07:05.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:07:05.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:07:05.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:07:05.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:07:05.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:07:05.896230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:07:05.897351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:07:05.897351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:07:05.897351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:07:05.897351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:07:05.897351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:07:05.897351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:07:05.897351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:07:05.897351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:07:05.897351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:07:05.897351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:07:05.897351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:07:05.897351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:07:05.897351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 595/1195] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:07:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:07:05.899975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd9d134-d62e-400b-9356-cef71bac1b83 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:07:05.924660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:07:06.002388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd9d134-d62e-400b-9356-cef71bac1b83 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:07:06.038124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2ea050-30f0-48af-b5ca-810dee6c1140 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:07:06.038733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2ea050-30f0-48af-b5ca-810dee6c1140 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:06.039012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2ea050-30f0-48af-b5ca-810dee6c1140 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:06.039284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2ea050-30f0-48af-b5ca-810dee6c1140 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:06.039967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2ea050-30f0-48af-b5ca-810dee6c1140 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:07:06.040300 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2ea050-30f0-48af-b5ca-810dee6c1140 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:07:06.041675 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2ea050-30f0-48af-b5ca-810dee6c1140 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:07:06.059124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 601/1196] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:07:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:07:06.066575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d863eb2e-9e6e-408b-9923-c31a9a2fdadb None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:07:06.172523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d863eb2e-9e6e-408b-9923-c31a9a2fdadb None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:07:06.206937 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66efd0df-900c-41cb-9e66-891514332cce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:07:06.207568 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66efd0df-900c-41cb-9e66-891514332cce None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:06.207842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66efd0df-900c-41cb-9e66-891514332cce None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:06.208134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66efd0df-900c-41cb-9e66-891514332cce None admin] PATH_INFO: `/v3/users/7971d60f7133455ba9504422b8e86438` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:06.209250 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66efd0df-900c-41cb-9e66-891514332cce None admin] RBAC: Authorizing `identity:get_user(user_id=7971d60f7133455ba9504422b8e86438)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:07:06.210668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66efd0df-900c-41cb-9e66-891514332cce None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:07:06.212332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 596/1197] 10.222.0.47 () {56 vars in 1177 bytes} [Sun Aug 21 01:07:06 2022] GET /identity/v3/users/7971d60f7133455ba9504422b8e86438 => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:07:06.221069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2ea050-30f0-48af-b5ca-810dee6c1140 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:07:06.308443 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:07:06.324009 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2ea050-30f0-48af-b5ca-810dee6c1140 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:07:06.363110 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17912e4-84f7-4b79-9fe3-c2d2003391a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:07:06.363829 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17912e4-84f7-4b79-9fe3-c2d2003391a3 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:06.364091 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17912e4-84f7-4b79-9fe3-c2d2003391a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:06.364375 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17912e4-84f7-4b79-9fe3-c2d2003391a3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:06.432261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17912e4-84f7-4b79-9fe3-c2d2003391a3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:07:06.433701 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17912e4-84f7-4b79-9fe3-c2d2003391a3 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:07:06.517789 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:07:06.534963 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 602/1198] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:07:06 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:07:06.541059 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66efd0df-900c-41cb-9e66-891514332cce None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:07:06.600190 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:07:06.647134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66efd0df-900c-41cb-9e66-891514332cce None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:07:06.684832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95620017-c71b-41ff-98d5-f6d5099dbe5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:07:06.685469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95620017-c71b-41ff-98d5-f6d5099dbe5f None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:06.685997 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95620017-c71b-41ff-98d5-f6d5099dbe5f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:06.686293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95620017-c71b-41ff-98d5-f6d5099dbe5f None admin] PATH_INFO: `/v3/projects/d77f6264687e403e8019bc11cc17aefe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:06.687433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95620017-c71b-41ff-98d5-f6d5099dbe5f None admin] RBAC: Authorizing `identity:get_project(project_id=d77f6264687e403e8019bc11cc17aefe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:07:06.688904 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95620017-c71b-41ff-98d5-f6d5099dbe5f None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:07:06.690593 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 597/1199] 10.222.0.47 () {56 vars in 1186 bytes} [Sun Aug 21 01:07:06 2022] GET /identity/v3/projects/d77f6264687e403e8019bc11cc17aefe => generated 361 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:07:06.697340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17912e4-84f7-4b79-9fe3-c2d2003391a3 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:07:06.799288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17912e4-84f7-4b79-9fe3-c2d2003391a3 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:07:06.839302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16273a71-8e95-4792-93f4-cfe8fe72db2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:07:06.839904 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16273a71-8e95-4792-93f4-cfe8fe72db2f None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:07:06.840229 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16273a71-8e95-4792-93f4-cfe8fe72db2f None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:07:06.840504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16273a71-8e95-4792-93f4-cfe8fe72db2f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:07:06.841220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16273a71-8e95-4792-93f4-cfe8fe72db2f None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=d77f6264687e403e8019bc11cc17aefe, user.id=7971d60f7133455ba9504422b8e86438) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:07:06.841526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16273a71-8e95-4792-93f4-cfe8fe72db2f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:07:06.842951 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16273a71-8e95-4792-93f4-cfe8fe72db2f None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:07:06.848776 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 603/1200] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:07:06 2022] GET /identity/v3/role_assignments?user.id=7971d60f7133455ba9504422b8e86438&scope.project.id=d77f6264687e403e8019bc11cc17aefe&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:07:16.680053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=39 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:07:16.846093 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=75 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:24.459032 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9e613d-a53f-4f56-941d-1e3e55ba0675 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:24.459926 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9e613d-a53f-4f56-941d-1e3e55ba0675 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:24.459926 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9e613d-a53f-4f56-941d-1e3e55ba0675 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:24.461581 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 598/1201] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:08:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:08:24.467211 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb7d65e-ac6c-43a7-8ec2-a93ab3e81fda None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:24.469586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb7d65e-ac6c-43a7-8ec2-a93ab3e81fda None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:24.469586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb7d65e-ac6c-43a7-8ec2-a93ab3e81fda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:24.487231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-3fb7d65e-ac6c-43a7-8ec2-a93ab3e81fda None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:08:24.488468 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-3fb7d65e-ac6c-43a7-8ec2-a93ab3e81fda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:08:24.561379 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 604/1202] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:08:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:08:24.829786 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9e613d-a53f-4f56-941d-1e3e55ba0675 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:24.937922 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9e613d-a53f-4f56-941d-1e3e55ba0675 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:24.985496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70edda9b-47a7-42f4-bbba-f45e928598f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:24.986458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70edda9b-47a7-42f4-bbba-f45e928598f1 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:24.986766 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70edda9b-47a7-42f4-bbba-f45e928598f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:24.987053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70edda9b-47a7-42f4-bbba-f45e928598f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:25.000863 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70edda9b-47a7-42f4-bbba-f45e928598f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:25.002313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70edda9b-47a7-42f4-bbba-f45e928598f1 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:25.016637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-70edda9b-47a7-42f4-bbba-f45e928598f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:25.016637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:25.016637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:25.016637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:25.016637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:25.016637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:25.016637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:25.016637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:25.016637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:25.016637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:25.016637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:25.016637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:25.016637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:08:25.016637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:08:25.016637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:08:25.016637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:25.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:08:25.020214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:08:25.020214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:25.020214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:08:25.020214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 599/1203] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:25.024240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb7d65e-ac6c-43a7-8ec2-a93ab3e81fda None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:25.129540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb7d65e-ac6c-43a7-8ec2-a93ab3e81fda None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:25.158686 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:25.164562 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf13a166-8a99-4998-bb74-4ec67b503a7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:25.165204 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf13a166-8a99-4998-bb74-4ec67b503a7d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:25.165448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf13a166-8a99-4998-bb74-4ec67b503a7d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:25.165714 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf13a166-8a99-4998-bb74-4ec67b503a7d None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:25.166341 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf13a166-8a99-4998-bb74-4ec67b503a7d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:25.166638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf13a166-8a99-4998-bb74-4ec67b503a7d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:25.167981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf13a166-8a99-4998-bb74-4ec67b503a7d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:25.179513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 605/1204] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:08:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:25.185636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70edda9b-47a7-42f4-bbba-f45e928598f1 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:25.239604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:25.285788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70edda9b-47a7-42f4-bbba-f45e928598f1 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:25.318855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd2ea17-34a3-44ae-adfb-93d17bc9d2ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:25.319454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd2ea17-34a3-44ae-adfb-93d17bc9d2ed None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:25.319729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd2ea17-34a3-44ae-adfb-93d17bc9d2ed None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:25.319997 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd2ea17-34a3-44ae-adfb-93d17bc9d2ed None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:25.320651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd2ea17-34a3-44ae-adfb-93d17bc9d2ed None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:25.322064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd2ea17-34a3-44ae-adfb-93d17bc9d2ed None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:25.552634 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 600/1205] 10.222.0.47 () {60 vars in 1132 bytes} [Sun Aug 21 01:08:25 2022] POST /identity/v3/users => generated 251 bytes in 367 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:08:27.483588 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698d6777-431b-4525-836c-2c30d9be7443 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:27.484173 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698d6777-431b-4525-836c-2c30d9be7443 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:27.484652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698d6777-431b-4525-836c-2c30d9be7443 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:27.487025 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 606/1206] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:08:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:08:27.492185 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb2d242-409e-4051-8c7c-2c0f02d06528 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:27.492711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb2d242-409e-4051-8c7c-2c0f02d06528 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:27.492842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb2d242-409e-4051-8c7c-2c0f02d06528 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:27.512093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-ceb2d242-409e-4051-8c7c-2c0f02d06528 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:08:27.513351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ceb2d242-409e-4051-8c7c-2c0f02d06528 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:08:27.581252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 601/1207] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:08:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:08:27.850160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698d6777-431b-4525-836c-2c30d9be7443 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:27.967166 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698d6777-431b-4525-836c-2c30d9be7443 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:28.008202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b013c11d-52f9-43f3-9446-afb36444b4da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:28.008854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b013c11d-52f9-43f3-9446-afb36444b4da None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:28.009109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b013c11d-52f9-43f3-9446-afb36444b4da None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:28.009374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b013c11d-52f9-43f3-9446-afb36444b4da None admin] PATH_INFO: `/v3/roles/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:28.020787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b013c11d-52f9-43f3-9446-afb36444b4da None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:08:28.027938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-b013c11d-52f9-43f3-9446-afb36444b4da None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:08:28.027938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:28.027938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:28.027938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:28.027938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:28.027938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:28.027938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:28.027938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:28.027938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:28.027938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:28.027938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:28.027938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:28.027938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:08:28.027938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:08:28.027938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:08:28.027938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:08:28.030080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:08:28.030080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:08:28.030080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:08:28.030080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:08:28.030080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:28.030080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:28.030080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:28.030080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:28.030080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:28.030080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:28.030080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:28.030080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:28.030080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:28.030080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:28.030080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:28.030080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:28.032694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:28.032694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:28.032694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:28.032694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:28.032694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:08:28.032694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:08:28.032694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:08:28.032694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:08:28.032694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:08:28.032694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:08:28.032694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:08:28.032694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:28.032694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 607/1208] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:08:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:28.036716 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb2d242-409e-4051-8c7c-2c0f02d06528 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:28.088803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:28.158300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb2d242-409e-4051-8c7c-2c0f02d06528 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:28.193670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829e2c85-3825-4ada-909a-dc0da932e123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:28.194288 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829e2c85-3825-4ada-909a-dc0da932e123 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:28.194560 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829e2c85-3825-4ada-909a-dc0da932e123 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:28.194828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829e2c85-3825-4ada-909a-dc0da932e123 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:28.195486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829e2c85-3825-4ada-909a-dc0da932e123 None admin] RBAC: Adding query filter params (name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:28.195786 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829e2c85-3825-4ada-909a-dc0da932e123 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:28.197207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829e2c85-3825-4ada-909a-dc0da932e123 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:28.208174 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 602/1209] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:08:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:08:28.213967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b013c11d-52f9-43f3-9446-afb36444b4da None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:28.271537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:28.330998 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b013c11d-52f9-43f3-9446-afb36444b4da None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:28.373159 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9161fa-c144-4e5f-8d06-201239f197c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:28.373782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9161fa-c144-4e5f-8d06-201239f197c4 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:28.374061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9161fa-c144-4e5f-8d06-201239f197c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:28.374826 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9161fa-c144-4e5f-8d06-201239f197c4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:28.392512 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9161fa-c144-4e5f-8d06-201239f197c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:28.393961 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9161fa-c144-4e5f-8d06-201239f197c4 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:28.413949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-6e9161fa-c144-4e5f-8d06-201239f197c4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:28.413949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:28.413949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:28.413949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:28.413949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:28.413949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:28.413949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:28.413949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:28.413949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:28.413949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:28.413949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:28.413949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:28.413949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:08:28.413949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:08:28.413949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:08:28.413949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:28.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:08:28.416494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:08:28.416494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:28.416494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:28.416494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 608/1210] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:08:28.416805 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829e2c85-3825-4ada-909a-dc0da932e123 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:28.532812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:28.535797 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829e2c85-3825-4ada-909a-dc0da932e123 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:28.571813 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c49b03-0166-4037-a238-676fb3647fee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:28.572444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c49b03-0166-4037-a238-676fb3647fee None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:28.572714 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c49b03-0166-4037-a238-676fb3647fee None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:28.572976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c49b03-0166-4037-a238-676fb3647fee None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:28.573607 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c49b03-0166-4037-a238-676fb3647fee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:28.573896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c49b03-0166-4037-a238-676fb3647fee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:28.575346 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c49b03-0166-4037-a238-676fb3647fee None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:28.587665 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 603/1211] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:08:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:28.594398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9161fa-c144-4e5f-8d06-201239f197c4 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:28.711754 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9161fa-c144-4e5f-8d06-201239f197c4 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:28.713435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:28.747353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a5dea4-9315-4cc1-ad39-3e86a265a488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:28.748020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a5dea4-9315-4cc1-ad39-3e86a265a488 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:28.748317 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a5dea4-9315-4cc1-ad39-3e86a265a488 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:28.748585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a5dea4-9315-4cc1-ad39-3e86a265a488 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:28.756629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a5dea4-9315-4cc1-ad39-3e86a265a488 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:28.758096 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a5dea4-9315-4cc1-ad39-3e86a265a488 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:28.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-70a5dea4-9315-4cc1-ad39-3e86a265a488 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 21 01:08:28.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:28.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:28.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:28.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:28.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:28.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:28.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:28.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:28.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:28.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:28.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:28.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:08:28.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:08:28.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:08:28.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:28.774400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:28.776659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:28.776659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:28.776659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:08:28.776659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:08:28.776659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:08:28.776659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:08:28.776659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:08:28.776659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:08:28.776659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 21 01:08:28.776659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:28.776659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 609/1212] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:28 2022] GET /identity/v3/users/placement => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:08:28.779940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c49b03-0166-4037-a238-676fb3647fee None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:28.893134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c49b03-0166-4037-a238-676fb3647fee None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:28.933452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467b1bdb-49d2-4c08-a8f6-7f042c8526ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:28.934072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467b1bdb-49d2-4c08-a8f6-7f042c8526ee None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:28.934343 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467b1bdb-49d2-4c08-a8f6-7f042c8526ee None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:28.934612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467b1bdb-49d2-4c08-a8f6-7f042c8526ee None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:28.935437 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467b1bdb-49d2-4c08-a8f6-7f042c8526ee None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:28.935736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467b1bdb-49d2-4c08-a8f6-7f042c8526ee None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:28.937334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467b1bdb-49d2-4c08-a8f6-7f042c8526ee None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:28.956209 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 604/1213] 10.222.0.47 () {56 vars in 1143 bytes} [Sun Aug 21 01:08:28 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:08:28.962353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a5dea4-9315-4cc1-ad39-3e86a265a488 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:29.069225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a5dea4-9315-4cc1-ad39-3e86a265a488 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:29.107853 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e095800-0f54-42ea-9e5b-eac89b9ac37d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:29.108493 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e095800-0f54-42ea-9e5b-eac89b9ac37d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:29.109389 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e095800-0f54-42ea-9e5b-eac89b9ac37d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:29.109659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e095800-0f54-42ea-9e5b-eac89b9ac37d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:29.141269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:29.188915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e095800-0f54-42ea-9e5b-eac89b9ac37d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:29.190421 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e095800-0f54-42ea-9e5b-eac89b9ac37d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:29.298039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 610/1214] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:08:28 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 21 01:08:29.303860 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467b1bdb-49d2-4c08-a8f6-7f042c8526ee None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:29.322690 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:29.417985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467b1bdb-49d2-4c08-a8f6-7f042c8526ee None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:29.459423 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea2e520-165d-4b73-8d14-13a23f6e4cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:29.460057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea2e520-165d-4b73-8d14-13a23f6e4cfc None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:29.460361 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea2e520-165d-4b73-8d14-13a23f6e4cfc None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:29.460632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea2e520-165d-4b73-8d14-13a23f6e4cfc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:29.471961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea2e520-165d-4b73-8d14-13a23f6e4cfc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:29.473571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea2e520-165d-4b73-8d14-13a23f6e4cfc None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:29.488462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-4ea2e520-165d-4b73-8d14-13a23f6e4cfc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:29.488462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:29.488462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:29.488462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:29.488462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:29.488462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:29.488462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:29.488462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:29.488462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:29.488462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:29.488462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:29.488462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:29.488462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:08:29.488462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:08:29.488462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:08:29.488462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:29.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:08:29.490469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:08:29.490469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:29.490469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:08:29.490469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 605/1215] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:29.495636 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e095800-0f54-42ea-9e5b-eac89b9ac37d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:29.559956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:29.600590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e095800-0f54-42ea-9e5b-eac89b9ac37d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:29.636327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2dd2d5-a0e1-4114-bcc8-d4d2314a0c35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:29.636952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2dd2d5-a0e1-4114-bcc8-d4d2314a0c35 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:29.637230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2dd2d5-a0e1-4114-bcc8-d4d2314a0c35 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:29.637504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2dd2d5-a0e1-4114-bcc8-d4d2314a0c35 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:29.638135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2dd2d5-a0e1-4114-bcc8-d4d2314a0c35 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:29.638491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2dd2d5-a0e1-4114-bcc8-d4d2314a0c35 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:29.639910 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2dd2d5-a0e1-4114-bcc8-d4d2314a0c35 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:29.652431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 611/1216] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:08:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:29.658507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea2e520-165d-4b73-8d14-13a23f6e4cfc None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:29.731975 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:29.763945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea2e520-165d-4b73-8d14-13a23f6e4cfc None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:29.797298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e35049-efe6-4deb-a972-77cf8d0ded4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:29.797921 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e35049-efe6-4deb-a972-77cf8d0ded4e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:29.798192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e35049-efe6-4deb-a972-77cf8d0ded4e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:29.798456 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e35049-efe6-4deb-a972-77cf8d0ded4e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:29.805164 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e35049-efe6-4deb-a972-77cf8d0ded4e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:29.806583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e35049-efe6-4deb-a972-77cf8d0ded4e None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:29.818310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-44e35049-efe6-4deb-a972-77cf8d0ded4e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:08:29.818310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:29.818310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:29.818310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:29.818310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:29.818310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:29.818310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:29.818310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:29.818310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:29.818310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:29.818310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:29.818310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:29.818310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:08:29.818310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:08:29.818310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:29.819318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:29.820578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:08:29.820578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:08:29.820578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:08:29.820578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:08:29.820578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:08:29.820578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:08:29.820578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:08:29.820578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:08:29.820578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 606/1217] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:08:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:29.826850 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2dd2d5-a0e1-4114-bcc8-d4d2314a0c35 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:29.943309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2dd2d5-a0e1-4114-bcc8-d4d2314a0c35 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:29.968515 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:29.979559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e4ba7d-2f2d-4b87-933e-6c529e4d6976 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:29.980173 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e4ba7d-2f2d-4b87-933e-6c529e4d6976 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:29.980453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e4ba7d-2f2d-4b87-933e-6c529e4d6976 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:29.980717 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e4ba7d-2f2d-4b87-933e-6c529e4d6976 None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:29.981387 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e4ba7d-2f2d-4b87-933e-6c529e4d6976 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:29.981709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e4ba7d-2f2d-4b87-933e-6c529e4d6976 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:29.983072 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e4ba7d-2f2d-4b87-933e-6c529e4d6976 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:29.994820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 612/1218] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:08:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:30.001962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e35049-efe6-4deb-a972-77cf8d0ded4e None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:30.112616 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e35049-efe6-4deb-a972-77cf8d0ded4e None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:30.143441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:30.149259 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21deb73-b1c7-4492-88d9-96f865e57fa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:30.149861 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21deb73-b1c7-4492-88d9-96f865e57fa5 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:30.150133 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21deb73-b1c7-4492-88d9-96f865e57fa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:30.150429 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21deb73-b1c7-4492-88d9-96f865e57fa5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:30.151109 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21deb73-b1c7-4492-88d9-96f865e57fa5 None admin] RBAC: Adding query filter params (role.id=c2d8bf7283e14857879d03d1f39c0165, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=972b0f1dbfef4d85a6dfc4c5fd839bde) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:30.151410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21deb73-b1c7-4492-88d9-96f865e57fa5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:30.152856 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21deb73-b1c7-4492-88d9-96f865e57fa5 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:30.159125 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 607/1219] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:08:30 2022] GET /identity/v3/role_assignments?user.id=972b0f1dbfef4d85a6dfc4c5fd839bde&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=c2d8bf7283e14857879d03d1f39c0165 => generated 259 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:32.056641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feb65af-0efe-4f62-a545-6d3cf5106809 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:32.057218 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feb65af-0efe-4f62-a545-6d3cf5106809 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:32.057218 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feb65af-0efe-4f62-a545-6d3cf5106809 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:32.058505 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 613/1220] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:08:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:08:32.065102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a76d5b-c52f-44f6-a21c-d66331ddeb6c None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:32.065102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a76d5b-c52f-44f6-a21c-d66331ddeb6c None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:32.066814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a76d5b-c52f-44f6-a21c-d66331ddeb6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:32.088621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-e7a76d5b-c52f-44f6-a21c-d66331ddeb6c None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:08:32.089851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e7a76d5b-c52f-44f6-a21c-d66331ddeb6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:08:32.162091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 608/1221] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:08:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:08:32.428347 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7feb65af-0efe-4f62-a545-6d3cf5106809 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:32.546001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7feb65af-0efe-4f62-a545-6d3cf5106809 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:32.582227 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc10316-e860-4d55-96ec-2e505d2bd739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:32.582854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc10316-e860-4d55-96ec-2e505d2bd739 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:32.583130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc10316-e860-4d55-96ec-2e505d2bd739 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:32.583400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc10316-e860-4d55-96ec-2e505d2bd739 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:32.593589 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc10316-e860-4d55-96ec-2e505d2bd739 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:08:32.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-acc10316-e860-4d55-96ec-2e505d2bd739 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:08:32.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:32.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:32.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:32.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:32.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:32.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:32.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:32.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:32.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:32.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:32.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:32.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:08:32.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:08:32.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:08:32.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:08:32.599474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:08:32.599474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:08:32.599474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:08:32.599474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:08:32.599474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:32.599474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:32.599474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:32.599474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:32.599474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:32.599474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:32.599474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:32.599474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:32.599474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:32.599474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:32.599474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:32.599474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:32.600527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:32.600527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:32.600527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:32.600527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:32.600527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:08:32.600527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:08:32.600527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:08:32.600527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:08:32.600527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:08:32.600527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:08:32.600527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:08:32.600527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:32.600527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 614/1222] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:08:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:32.605567 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a76d5b-c52f-44f6-a21c-d66331ddeb6c None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:32.722287 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a76d5b-c52f-44f6-a21c-d66331ddeb6c None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:32.746205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:32.757130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1710560a-d0b5-46c6-af81-7b2c7c3dbfa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:32.757729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1710560a-d0b5-46c6-af81-7b2c7c3dbfa2 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:32.758016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1710560a-d0b5-46c6-af81-7b2c7c3dbfa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:32.758267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1710560a-d0b5-46c6-af81-7b2c7c3dbfa2 None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:32.758916 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1710560a-d0b5-46c6-af81-7b2c7c3dbfa2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:32.759207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1710560a-d0b5-46c6-af81-7b2c7c3dbfa2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:32.760585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1710560a-d0b5-46c6-af81-7b2c7c3dbfa2 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:32.769469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 609/1223] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:08:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:32.778187 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc10316-e860-4d55-96ec-2e505d2bd739 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:32.825292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:32.888025 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc10316-e860-4d55-96ec-2e505d2bd739 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:32.925102 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43a8c03-eb68-4644-becd-b1e345dad285 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:32.925728 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43a8c03-eb68-4644-becd-b1e345dad285 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:32.925999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43a8c03-eb68-4644-becd-b1e345dad285 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:32.926264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43a8c03-eb68-4644-becd-b1e345dad285 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:32.937640 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43a8c03-eb68-4644-becd-b1e345dad285 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:32.939082 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43a8c03-eb68-4644-becd-b1e345dad285 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:32.953716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-e43a8c03-eb68-4644-becd-b1e345dad285 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:32.953716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:32.953716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:32.953716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:32.953716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:32.953716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:32.953716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:32.953716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:32.953716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:32.953716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:32.953716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:32.953716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:32.953716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:08:32.953716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:08:32.953716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:08:32.953716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:32.954965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:08:32.955917 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:08:32.955917 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:32.955917 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:32.955917 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 615/1224] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:32.961752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1710560a-d0b5-46c6-af81-7b2c7c3dbfa2 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:33.071434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1710560a-d0b5-46c6-af81-7b2c7c3dbfa2 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:33.111526 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e8a166-bd6c-4696-a92d-6253904075e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:33.112156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e8a166-bd6c-4696-a92d-6253904075e5 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:33.112436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e8a166-bd6c-4696-a92d-6253904075e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:33.112704 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e8a166-bd6c-4696-a92d-6253904075e5 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:33.113350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e8a166-bd6c-4696-a92d-6253904075e5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:33.113647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e8a166-bd6c-4696-a92d-6253904075e5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:33.115060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e8a166-bd6c-4696-a92d-6253904075e5 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:33.128617 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 610/1225] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:08:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:08:33.134388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43a8c03-eb68-4644-becd-b1e345dad285 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:33.231465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:33.243234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43a8c03-eb68-4644-becd-b1e345dad285 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:33.277242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a4eaf2-6abf-423c-aa30-bb6dbac5b83b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:33.279652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a4eaf2-6abf-423c-aa30-bb6dbac5b83b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:33.279652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a4eaf2-6abf-423c-aa30-bb6dbac5b83b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:33.279652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a4eaf2-6abf-423c-aa30-bb6dbac5b83b None admin] PATH_INFO: `/v3/users/placement` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:33.289078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a4eaf2-6abf-423c-aa30-bb6dbac5b83b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:33.291737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a4eaf2-6abf-423c-aa30-bb6dbac5b83b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:33.304787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-17a4eaf2-6abf-423c-aa30-bb6dbac5b83b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 21 01:08:33.304787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:33.304787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:33.304787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:33.304787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:33.304787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:33.304787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:33.304787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:33.304787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:33.304787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:33.304787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:33.304787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:33.304787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:08:33.304787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:08:33.304787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:08:33.304787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:33.306658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:33.313240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:33.313240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:33.313240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:08:33.313240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:08:33.313240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:08:33.313240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:08:33.313240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:08:33.313240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:08:33.313240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 21 01:08:33.313240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:33.313240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 616/1226] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:33 2022] GET /identity/v3/users/placement => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:33.314164 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e8a166-bd6c-4696-a92d-6253904075e5 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:33.362686 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:33.414819 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e8a166-bd6c-4696-a92d-6253904075e5 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:33.456745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd252e5-6eb7-477e-b43c-351d32fb9410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:33.457352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd252e5-6eb7-477e-b43c-351d32fb9410 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:33.457626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd252e5-6eb7-477e-b43c-351d32fb9410 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:33.457893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd252e5-6eb7-477e-b43c-351d32fb9410 None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:33.458614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd252e5-6eb7-477e-b43c-351d32fb9410 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:33.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd252e5-6eb7-477e-b43c-351d32fb9410 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:33.460436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd252e5-6eb7-477e-b43c-351d32fb9410 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:33.489774 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 611/1227] 10.222.0.47 () {56 vars in 1143 bytes} [Sun Aug 21 01:08:33 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:08:33.495835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a4eaf2-6abf-423c-aa30-bb6dbac5b83b None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:33.610375 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a4eaf2-6abf-423c-aa30-bb6dbac5b83b None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:33.647078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bcd536-cfac-460b-ac20-1438252dbdd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:33.647712 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bcd536-cfac-460b-ac20-1438252dbdd7 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:33.647988 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bcd536-cfac-460b-ac20-1438252dbdd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:33.648289 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bcd536-cfac-460b-ac20-1438252dbdd7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:33.659772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bcd536-cfac-460b-ac20-1438252dbdd7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:33.663244 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bcd536-cfac-460b-ac20-1438252dbdd7 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:33.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-84bcd536-cfac-460b-ac20-1438252dbdd7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:33.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:33.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:33.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:33.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:33.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:33.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:33.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:33.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:33.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:33.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:33.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:33.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:08:33.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:08:33.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:08:33.677215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:33.678540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:08:33.680243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:08:33.680243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:33.680243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:33.680243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 617/1228] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:08:33.686196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd252e5-6eb7-477e-b43c-351d32fb9410 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:33.797636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd252e5-6eb7-477e-b43c-351d32fb9410 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:33.799959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:33.833677 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eff820-a0bb-41ac-9d51-61cef88bfa2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:33.835455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eff820-a0bb-41ac-9d51-61cef88bfa2e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:33.835455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eff820-a0bb-41ac-9d51-61cef88bfa2e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:33.835455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eff820-a0bb-41ac-9d51-61cef88bfa2e None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:33.835734 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29eff820-a0bb-41ac-9d51-61cef88bfa2e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:33.835865 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29eff820-a0bb-41ac-9d51-61cef88bfa2e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:33.837275 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29eff820-a0bb-41ac-9d51-61cef88bfa2e None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:33.849573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 612/1229] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:08:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:33.856405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bcd536-cfac-460b-ac20-1438252dbdd7 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:33.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:33.964357 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bcd536-cfac-460b-ac20-1438252dbdd7 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:34.002937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80028ebb-de9a-4310-ba39-435bd5a24e4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:34.003566 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80028ebb-de9a-4310-ba39-435bd5a24e4f None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:34.003924 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80028ebb-de9a-4310-ba39-435bd5a24e4f None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:34.004224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80028ebb-de9a-4310-ba39-435bd5a24e4f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:34.011507 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80028ebb-de9a-4310-ba39-435bd5a24e4f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:34.012980 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80028ebb-de9a-4310-ba39-435bd5a24e4f None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:34.025384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-80028ebb-de9a-4310-ba39-435bd5a24e4f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:08:34.025384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:34.025384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:34.025384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:34.025384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:34.025384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:34.025384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:34.025384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:34.025384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:34.025384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:34.025384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:34.025384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:34.025384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:08:34.025384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:08:34.025384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:34.026568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:34.027812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:08:34.027812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:08:34.027812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:08:34.027812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:08:34.027812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:08:34.027812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:08:34.027812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:08:34.027812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:34.027812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 618/1230] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:08:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:34.033454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eff820-a0bb-41ac-9d51-61cef88bfa2e None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:34.147473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eff820-a0bb-41ac-9d51-61cef88bfa2e None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:34.181960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e741ef-f084-4f49-bbf0-df348e08bbd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:34.182564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e741ef-f084-4f49-bbf0-df348e08bbd6 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:34.182839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e741ef-f084-4f49-bbf0-df348e08bbd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:34.183104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e741ef-f084-4f49-bbf0-df348e08bbd6 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:34.183785 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e741ef-f084-4f49-bbf0-df348e08bbd6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:34.184084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e741ef-f084-4f49-bbf0-df348e08bbd6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:34.185533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e741ef-f084-4f49-bbf0-df348e08bbd6 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:34.198077 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 613/1231] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:08:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:34.206122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80028ebb-de9a-4310-ba39-435bd5a24e4f None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:34.282510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:34.317617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80028ebb-de9a-4310-ba39-435bd5a24e4f None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:34.355334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c52d68b-a95c-4ae6-9475-a710f79c2c72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:34.356123 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c52d68b-a95c-4ae6-9475-a710f79c2c72 None admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:34.356415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c52d68b-a95c-4ae6-9475-a710f79c2c72 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:34.356689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c52d68b-a95c-4ae6-9475-a710f79c2c72 None admin] PATH_INFO: `/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/972b0f1dbfef4d85a6dfc4c5fd839bde/roles/c2d8bf7283e14857879d03d1f39c0165` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:34.379687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c52d68b-a95c-4ae6-9475-a710f79c2c72 None admin] RBAC: Authorizing `identity:create_grant(project_id=75a3bf3f3a4244d18fde47d492657db4, user_id=972b0f1dbfef4d85a6dfc4c5fd839bde, role_id=c2d8bf7283e14857879d03d1f39c0165)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:34.381191 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c52d68b-a95c-4ae6-9475-a710f79c2c72 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:34.606747 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 619/1232] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:08:34 2022] PUT /identity/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/972b0f1dbfef4d85a6dfc4c5fd839bde/roles/c2d8bf7283e14857879d03d1f39c0165 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:08:36.527695 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9e902d-8be1-4f9a-b4a8-955a57a710b9 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:36.528293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9e902d-8be1-4f9a-b4a8-955a57a710b9 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:36.528513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9e902d-8be1-4f9a-b4a8-955a57a710b9 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:36.529962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 614/1233] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:08:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:08:36.535132 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae0182d-1c9b-44e2-aece-5e993dfd8615 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:36.535526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae0182d-1c9b-44e2-aece-5e993dfd8615 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:36.535687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae0182d-1c9b-44e2-aece-5e993dfd8615 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:36.553491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-bae0182d-1c9b-44e2-aece-5e993dfd8615 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:08:36.554945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-bae0182d-1c9b-44e2-aece-5e993dfd8615 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:08:36.622440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 620/1234] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:08:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:08:36.898183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9e902d-8be1-4f9a-b4a8-955a57a710b9 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:36.924282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:37.004410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9e902d-8be1-4f9a-b4a8-955a57a710b9 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:37.041194 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ddbbc1-adf0-463a-82f6-485d9e6659b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:37.041812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ddbbc1-adf0-463a-82f6-485d9e6659b3 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:37.042083 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ddbbc1-adf0-463a-82f6-485d9e6659b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:37.042347 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ddbbc1-adf0-463a-82f6-485d9e6659b3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:37.053388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ddbbc1-adf0-463a-82f6-485d9e6659b3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:08:37.058600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-52ddbbc1-adf0-463a-82f6-485d9e6659b3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:08:37.058600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:37.058600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:37.058600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:37.058600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:37.058600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:37.058600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:37.058600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:37.058600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:37.058600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:37.058600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:37.058600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:37.058600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:08:37.058600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:08:37.058600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:08:37.058600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:08:37.059752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:08:37.059752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:08:37.059752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:08:37.059752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:08:37.059752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:37.059752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:37.059752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:37.059752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:37.059752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:37.059752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:37.059752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:37.059752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:37.059752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:37.059752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:37.059752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:37.059752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:37.060812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:37.060812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:37.060812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:37.060812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:37.060812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:08:37.060812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:08:37.060812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:08:37.060812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:08:37.060812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:08:37.060812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:08:37.060812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 21 01:08:37.060812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:08:37.060812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 615/1235] 10.222.0.47 () {56 vars in 1102 bytes} [Sun Aug 21 01:08:36 2022] GET /identity/v3/roles/service => generated 85 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:37.066028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae0182d-1c9b-44e2-aece-5e993dfd8615 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:37.113450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:37.167745 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae0182d-1c9b-44e2-aece-5e993dfd8615 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:37.205156 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824a0bb6-833f-4eda-9c4e-63ecdb03b41d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:37.205770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824a0bb6-833f-4eda-9c4e-63ecdb03b41d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:37.206048 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824a0bb6-833f-4eda-9c4e-63ecdb03b41d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:37.207374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824a0bb6-833f-4eda-9c4e-63ecdb03b41d None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:37.207374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824a0bb6-833f-4eda-9c4e-63ecdb03b41d None admin] RBAC: Adding query filter params (name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:37.207374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824a0bb6-833f-4eda-9c4e-63ecdb03b41d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:37.208805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824a0bb6-833f-4eda-9c4e-63ecdb03b41d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:37.218203 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 621/1236] 10.222.0.47 () {56 vars in 1103 bytes} [Sun Aug 21 01:08:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:37.224311 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ddbbc1-adf0-463a-82f6-485d9e6659b3 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:37.316007 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:37.331587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ddbbc1-adf0-463a-82f6-485d9e6659b3 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:37.366839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a135917c-b4b5-4191-9dd1-0f1a6e8d158b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:37.367460 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a135917c-b4b5-4191-9dd1-0f1a6e8d158b None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:37.367731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a135917c-b4b5-4191-9dd1-0f1a6e8d158b None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:37.367995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a135917c-b4b5-4191-9dd1-0f1a6e8d158b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:37.379368 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a135917c-b4b5-4191-9dd1-0f1a6e8d158b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:37.380821 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a135917c-b4b5-4191-9dd1-0f1a6e8d158b None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:37.395803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-a135917c-b4b5-4191-9dd1-0f1a6e8d158b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:37.395803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:37.395803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:37.395803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:37.395803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:37.395803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:37.395803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:37.395803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:37.395803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:37.395803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:37.395803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:37.395803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:37.395803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:08:37.395803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:08:37.395803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:08:37.395803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:37.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:08:37.398295 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:08:37.398295 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:37.398295 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:08:37.398295 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 616/1237] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:37.402905 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824a0bb6-833f-4eda-9c4e-63ecdb03b41d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:37.519216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824a0bb6-833f-4eda-9c4e-63ecdb03b41d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:37.528862 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:37.554751 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5e99e0-4426-45a0-a777-dc714a0723d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:37.555683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5e99e0-4426-45a0-a777-dc714a0723d5 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:37.555683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5e99e0-4426-45a0-a777-dc714a0723d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:37.555923 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5e99e0-4426-45a0-a777-dc714a0723d5 None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:37.556560 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5e99e0-4426-45a0-a777-dc714a0723d5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:37.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5e99e0-4426-45a0-a777-dc714a0723d5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:37.558224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5e99e0-4426-45a0-a777-dc714a0723d5 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:37.570286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 622/1238] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:08:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:37.576769 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a135917c-b4b5-4191-9dd1-0f1a6e8d158b None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:37.687296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a135917c-b4b5-4191-9dd1-0f1a6e8d158b None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:37.723235 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8babcab-ac2a-4824-aeb0-02ae49bfea0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:37.723896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8babcab-ac2a-4824-aeb0-02ae49bfea0e None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:37.724218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8babcab-ac2a-4824-aeb0-02ae49bfea0e None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:37.724489 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8babcab-ac2a-4824-aeb0-02ae49bfea0e None admin] PATH_INFO: `/v3/users/placement` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:37.727485 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:37.732763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8babcab-ac2a-4824-aeb0-02ae49bfea0e None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:37.734198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8babcab-ac2a-4824-aeb0-02ae49bfea0e None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:37.746061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-c8babcab-ac2a-4824-aeb0-02ae49bfea0e None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 21 01:08:37.746061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:37.746061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:37.746061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:37.746061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:37.746061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:37.746061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:37.746061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:37.746061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:37.746061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:37.746061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:37.746061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:37.746061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:08:37.746061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:08:37.746061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:08:37.746061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:37.750406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:37.751592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:37.751592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:37.751592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:08:37.751592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:08:37.751592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:08:37.751592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:08:37.751592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:08:37.751592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:08:37.751592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 21 01:08:37.751592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:08:37.751592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 617/1239] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:37 2022] GET /identity/v3/users/placement => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:37.753344 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5e99e0-4426-45a0-a777-dc714a0723d5 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:37.862927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5e99e0-4426-45a0-a777-dc714a0723d5 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:37.899228 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420e86a1-d447-413b-acd5-7542029b9a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:37.899884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420e86a1-d447-413b-acd5-7542029b9a35 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:37.900187 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420e86a1-d447-413b-acd5-7542029b9a35 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:37.900462 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420e86a1-d447-413b-acd5-7542029b9a35 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:37.901426 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420e86a1-d447-413b-acd5-7542029b9a35 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:37.901756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420e86a1-d447-413b-acd5-7542029b9a35 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:37.903174 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420e86a1-d447-413b-acd5-7542029b9a35 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:37.921089 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 623/1240] 10.222.0.47 () {56 vars in 1143 bytes} [Sun Aug 21 01:08:37 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:37.927204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8babcab-ac2a-4824-aeb0-02ae49bfea0e None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:38.036471 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8babcab-ac2a-4824-aeb0-02ae49bfea0e None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:38.071797 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce707221-7ba5-45e1-a5d2-802bfb8cb8aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:38.072551 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce707221-7ba5-45e1-a5d2-802bfb8cb8aa None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:38.072821 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce707221-7ba5-45e1-a5d2-802bfb8cb8aa None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:38.073087 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce707221-7ba5-45e1-a5d2-802bfb8cb8aa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:38.136526 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:38.143869 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce707221-7ba5-45e1-a5d2-802bfb8cb8aa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:38.145350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce707221-7ba5-45e1-a5d2-802bfb8cb8aa None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:38.255257 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 618/1241] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:08:37 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:08:38.262582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420e86a1-d447-413b-acd5-7542029b9a35 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:38.290220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:38.367085 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420e86a1-d447-413b-acd5-7542029b9a35 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:38.411240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d41f41e-06bc-46a8-85c7-1b4d7cc4101b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:38.412083 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d41f41e-06bc-46a8-85c7-1b4d7cc4101b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:38.412888 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d41f41e-06bc-46a8-85c7-1b4d7cc4101b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:38.412888 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d41f41e-06bc-46a8-85c7-1b4d7cc4101b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:38.425190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d41f41e-06bc-46a8-85c7-1b4d7cc4101b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:38.426659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d41f41e-06bc-46a8-85c7-1b4d7cc4101b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:38.444397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-4d41f41e-06bc-46a8-85c7-1b4d7cc4101b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:38.444397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:38.444397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:38.444397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:38.444397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:38.444397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:38.444397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:38.444397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:38.444397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:38.444397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:38.444397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:38.444397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:38.444397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:08:38.444397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:08:38.444397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:08:38.444397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:38.446222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:08:38.448053 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:08:38.448053 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:38.448053 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:38.448053 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 624/1242] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:38.451360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce707221-7ba5-45e1-a5d2-802bfb8cb8aa None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:38.580894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:38.589429 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce707221-7ba5-45e1-a5d2-802bfb8cb8aa None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:38.627336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a64ef4-0f2a-4204-acd1-2ab4aa80a9a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:38.627948 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a64ef4-0f2a-4204-acd1-2ab4aa80a9a7 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:38.628264 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a64ef4-0f2a-4204-acd1-2ab4aa80a9a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:38.628532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a64ef4-0f2a-4204-acd1-2ab4aa80a9a7 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:38.629172 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a64ef4-0f2a-4204-acd1-2ab4aa80a9a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:38.629472 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a64ef4-0f2a-4204-acd1-2ab4aa80a9a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:38.630844 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a64ef4-0f2a-4204-acd1-2ab4aa80a9a7 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:38.645939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 619/1243] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:08:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:38.651778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d41f41e-06bc-46a8-85c7-1b4d7cc4101b None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:38.735287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:38.756530 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d41f41e-06bc-46a8-85c7-1b4d7cc4101b None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:38.796642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d3a04c-690c-4185-ab27-6bcefb522964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:38.797650 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d3a04c-690c-4185-ab27-6bcefb522964 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:38.798099 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d3a04c-690c-4185-ab27-6bcefb522964 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:38.798531 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d3a04c-690c-4185-ab27-6bcefb522964 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:38.810437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d3a04c-690c-4185-ab27-6bcefb522964 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:38.812993 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d3a04c-690c-4185-ab27-6bcefb522964 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:38.832360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-c7d3a04c-690c-4185-ab27-6bcefb522964 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:08:38.832360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:38.832360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:38.832360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:38.832360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:38.832360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:38.832360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:38.832360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:38.832360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:38.832360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:38.832360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:38.832360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:38.832360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:08:38.832360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:08:38.832360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:38.834312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:38.835433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:08:38.835433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:08:38.835433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:08:38.835433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:08:38.835433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:08:38.835433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:08:38.835433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:08:38.835433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:38.835433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 625/1244] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:08:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:38.840417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a64ef4-0f2a-4204-acd1-2ab4aa80a9a7 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:38.950335 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a64ef4-0f2a-4204-acd1-2ab4aa80a9a7 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:38.986652 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2784919-3a1b-418a-b422-2bb18eb6e9d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:38.987174 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2784919-3a1b-418a-b422-2bb18eb6e9d7 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:38.987451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2784919-3a1b-418a-b422-2bb18eb6e9d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:38.987711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2784919-3a1b-418a-b422-2bb18eb6e9d7 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:38.988422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2784919-3a1b-418a-b422-2bb18eb6e9d7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:38.988720 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2784919-3a1b-418a-b422-2bb18eb6e9d7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:38.990136 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2784919-3a1b-418a-b422-2bb18eb6e9d7 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:39.003007 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 620/1245] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:08:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:39.010007 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d3a04c-690c-4185-ab27-6bcefb522964 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:39.126871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d3a04c-690c-4185-ab27-6bcefb522964 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:39.162319 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0195d8-4103-48eb-afc4-60a4909fad46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:39.162912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0195d8-4103-48eb-afc4-60a4909fad46 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:39.163473 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0195d8-4103-48eb-afc4-60a4909fad46 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:39.163473 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0195d8-4103-48eb-afc4-60a4909fad46 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:39.164193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0195d8-4103-48eb-afc4-60a4909fad46 None admin] RBAC: Adding query filter params (role.id=c2d8bf7283e14857879d03d1f39c0165, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=972b0f1dbfef4d85a6dfc4c5fd839bde) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:39.164497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0195d8-4103-48eb-afc4-60a4909fad46 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:39.165899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0195d8-4103-48eb-afc4-60a4909fad46 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:39.172663 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 626/1246] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:08:39 2022] GET /identity/v3/role_assignments?user.id=972b0f1dbfef4d85a6dfc4c5fd839bde&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=c2d8bf7283e14857879d03d1f39c0165 => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:41.157140 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b0613f-86ea-4001-b39a-6c941673c251 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:41.157958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b0613f-86ea-4001-b39a-6c941673c251 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:41.157958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b0613f-86ea-4001-b39a-6c941673c251 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:41.159099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 621/1247] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:08:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:08:41.165817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c39339b-b5ba-4b2c-9049-434c7872c123 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:41.166628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c39339b-b5ba-4b2c-9049-434c7872c123 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:41.166628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c39339b-b5ba-4b2c-9049-434c7872c123 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:41.173664 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:41.188734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-1c39339b-b5ba-4b2c-9049-434c7872c123 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:08:41.199009 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-1c39339b-b5ba-4b2c-9049-434c7872c123 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:08:41.272179 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 627/1248] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:08:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 21 01:08:41.557332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b0613f-86ea-4001-b39a-6c941673c251 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:41.577108 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:41.666231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b0613f-86ea-4001-b39a-6c941673c251 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:41.707973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e86e195-7536-4a07-aa60-99e02598b646 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:41.708635 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e86e195-7536-4a07-aa60-99e02598b646 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:41.708908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e86e195-7536-4a07-aa60-99e02598b646 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:41.709176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e86e195-7536-4a07-aa60-99e02598b646 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:41.720915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e86e195-7536-4a07-aa60-99e02598b646 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:08:41.725645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-0e86e195-7536-4a07-aa60-99e02598b646 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:08:41.725645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:41.725645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:41.725645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:41.725645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:41.725645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:41.725645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:41.725645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:41.725645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:41.725645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:41.725645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:41.725645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:41.725645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:08:41.725645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:08:41.725645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:08:41.725645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:08:41.727995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:08:41.727995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:08:41.727995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:08:41.727995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:08:41.727995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:41.727995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:41.727995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:41.727995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:41.727995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:41.727995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:41.727995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:41.727995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:41.727995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:41.727995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:41.727995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:41.727995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:41.729091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:41.729091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:41.729091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:41.729091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:41.729091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:08:41.729091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:08:41.729091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:08:41.729091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:08:41.729091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:08:41.729091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:08:41.729091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:08:41.729091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:08:41.729091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 622/1249] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:08:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:41.733116 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c39339b-b5ba-4b2c-9049-434c7872c123 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:41.832583 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c39339b-b5ba-4b2c-9049-434c7872c123 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:41.861695 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:41.866627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c297a7-8a0a-4126-afed-f61e529dfa39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:41.871368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c297a7-8a0a-4126-afed-f61e529dfa39 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:41.871368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c297a7-8a0a-4126-afed-f61e529dfa39 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:41.871368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c297a7-8a0a-4126-afed-f61e529dfa39 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:41.871368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c297a7-8a0a-4126-afed-f61e529dfa39 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:41.871368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c297a7-8a0a-4126-afed-f61e529dfa39 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:41.871368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c297a7-8a0a-4126-afed-f61e529dfa39 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:41.878739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 628/1250] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:08:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:41.884856 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e86e195-7536-4a07-aa60-99e02598b646 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:41.966022 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:41.990237 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e86e195-7536-4a07-aa60-99e02598b646 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:42.026132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b612c428-edbc-4c96-b275-41c58716ff6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:42.026757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b612c428-edbc-4c96-b275-41c58716ff6a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:42.027028 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b612c428-edbc-4c96-b275-41c58716ff6a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:42.027291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b612c428-edbc-4c96-b275-41c58716ff6a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:42.038367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b612c428-edbc-4c96-b275-41c58716ff6a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:42.039827 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b612c428-edbc-4c96-b275-41c58716ff6a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:42.055165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-b612c428-edbc-4c96-b275-41c58716ff6a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:42.055165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:42.055165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:42.055165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:42.055165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:42.055165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:42.055165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:42.055165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:42.055165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:42.055165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:42.055165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:42.055165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:42.055165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:08:42.055165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:08:42.055165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:08:42.055165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:42.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:08:42.057398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:08:42.057398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:42.057398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:08:42.057398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 623/1251] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:42.062950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c297a7-8a0a-4126-afed-f61e529dfa39 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:42.163164 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c297a7-8a0a-4126-afed-f61e529dfa39 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:42.200044 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0f893c-d7ae-4bd1-8c1c-7e68c1fbfb9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:42.200744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0f893c-d7ae-4bd1-8c1c-7e68c1fbfb9d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:42.200995 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0f893c-d7ae-4bd1-8c1c-7e68c1fbfb9d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:42.201264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0f893c-d7ae-4bd1-8c1c-7e68c1fbfb9d None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:42.201903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0f893c-d7ae-4bd1-8c1c-7e68c1fbfb9d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:42.202203 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0f893c-d7ae-4bd1-8c1c-7e68c1fbfb9d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:42.203618 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0f893c-d7ae-4bd1-8c1c-7e68c1fbfb9d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:42.215812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 629/1252] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:08:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:42.221821 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b612c428-edbc-4c96-b275-41c58716ff6a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:42.325647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b612c428-edbc-4c96-b275-41c58716ff6a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:42.352096 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:42.360630 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384a26c4-7a99-495a-9fb2-5bc4f8a0dd89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:42.361260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384a26c4-7a99-495a-9fb2-5bc4f8a0dd89 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:42.361534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384a26c4-7a99-495a-9fb2-5bc4f8a0dd89 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:42.361800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384a26c4-7a99-495a-9fb2-5bc4f8a0dd89 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:42.369469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384a26c4-7a99-495a-9fb2-5bc4f8a0dd89 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:42.370891 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384a26c4-7a99-495a-9fb2-5bc4f8a0dd89 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:42.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-384a26c4-7a99-495a-9fb2-5bc4f8a0dd89 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 21 01:08:42.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:42.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:42.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:42.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:42.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:42.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:42.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:42.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:42.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:42.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:42.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:42.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:08:42.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:08:42.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:08:42.382159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:42.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:42.384425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:42.384425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:42.384425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:08:42.384425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:08:42.384425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:08:42.384425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:08:42.384425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:08:42.384425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:08:42.384425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 21 01:08:42.384425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:08:42.384425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 624/1253] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:42 2022] GET /identity/v3/users/placement => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:08:42.389391 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0f893c-d7ae-4bd1-8c1c-7e68c1fbfb9d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:42.447768 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:42.492937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0f893c-d7ae-4bd1-8c1c-7e68c1fbfb9d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:42.526766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc67bb5b-13ca-4923-96f9-c63751dd5ca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:42.527362 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc67bb5b-13ca-4923-96f9-c63751dd5ca6 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:42.527636 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc67bb5b-13ca-4923-96f9-c63751dd5ca6 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:42.527922 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc67bb5b-13ca-4923-96f9-c63751dd5ca6 None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:42.528643 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc67bb5b-13ca-4923-96f9-c63751dd5ca6 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:42.529481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc67bb5b-13ca-4923-96f9-c63751dd5ca6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:42.530876 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc67bb5b-13ca-4923-96f9-c63751dd5ca6 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:42.549030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 630/1254] 10.222.0.47 () {56 vars in 1143 bytes} [Sun Aug 21 01:08:42 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:42.554946 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384a26c4-7a99-495a-9fb2-5bc4f8a0dd89 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:42.665057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384a26c4-7a99-495a-9fb2-5bc4f8a0dd89 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:42.702649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffc1e07-702d-4c17-80cc-d06721b8514f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:42.703256 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffc1e07-702d-4c17-80cc-d06721b8514f None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:42.703555 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffc1e07-702d-4c17-80cc-d06721b8514f None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:42.703822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffc1e07-702d-4c17-80cc-d06721b8514f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:42.759349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:42.771874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffc1e07-702d-4c17-80cc-d06721b8514f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:42.773390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffc1e07-702d-4c17-80cc-d06721b8514f None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:42.872978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 625/1255] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:08:42 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 21 01:08:42.879051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc67bb5b-13ca-4923-96f9-c63751dd5ca6 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:42.979726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc67bb5b-13ca-4923-96f9-c63751dd5ca6 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:42.992551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:43.012586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47dc605-28a2-4714-8d4a-8a2e216be135 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:43.013727 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47dc605-28a2-4714-8d4a-8a2e216be135 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:43.014003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47dc605-28a2-4714-8d4a-8a2e216be135 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:43.014267 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47dc605-28a2-4714-8d4a-8a2e216be135 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:43.023831 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47dc605-28a2-4714-8d4a-8a2e216be135 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:43.025302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47dc605-28a2-4714-8d4a-8a2e216be135 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:43.039907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-c47dc605-28a2-4714-8d4a-8a2e216be135 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:43.039907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:43.039907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:43.039907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:43.039907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:43.039907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:43.039907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:43.039907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:43.039907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:43.039907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:43.039907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:43.039907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:43.039907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:08:43.039907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:08:43.039907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:08:43.039907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:43.041432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:08:43.042684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:08:43.042684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:43.042684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:43.042684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 631/1256] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:43.047126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffc1e07-702d-4c17-80cc-d06721b8514f None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:43.152363 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:43.164875 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffc1e07-702d-4c17-80cc-d06721b8514f None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:43.203055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a68aee-0765-470c-9d63-5004901fcebf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:43.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a68aee-0765-470c-9d63-5004901fcebf None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:43.212388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a68aee-0765-470c-9d63-5004901fcebf None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:43.212388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a68aee-0765-470c-9d63-5004901fcebf None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:43.212388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a68aee-0765-470c-9d63-5004901fcebf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:43.212388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a68aee-0765-470c-9d63-5004901fcebf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:43.212388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a68aee-0765-470c-9d63-5004901fcebf None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:43.222093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 626/1257] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:08:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:43.227720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47dc605-28a2-4714-8d4a-8a2e216be135 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:43.351187 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47dc605-28a2-4714-8d4a-8a2e216be135 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:43.391266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be15a02-6b7d-46c3-9033-c5f978fa4787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:43.391921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be15a02-6b7d-46c3-9033-c5f978fa4787 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:43.392233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be15a02-6b7d-46c3-9033-c5f978fa4787 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:43.392507 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be15a02-6b7d-46c3-9033-c5f978fa4787 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:43.399825 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be15a02-6b7d-46c3-9033-c5f978fa4787 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:43.401316 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be15a02-6b7d-46c3-9033-c5f978fa4787 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:43.412893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-0be15a02-6b7d-46c3-9033-c5f978fa4787 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:08:43.412893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:43.412893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:43.412893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:43.412893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:43.412893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:43.412893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:43.412893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:43.412893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:43.412893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:43.412893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:43.412893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:43.412893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:08:43.412893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:08:43.412893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:43.414962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:08:43.414962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:08:43.414962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:08:43.414962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:08:43.414962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:08:43.414962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:08:43.414962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:08:43.414962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:43.414962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 632/1258] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:08:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:43.421040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a68aee-0765-470c-9d63-5004901fcebf None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:43.528807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a68aee-0765-470c-9d63-5004901fcebf None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:43.566599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d91de3-8024-4b42-8ab9-78e5170eb674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:43.567197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d91de3-8024-4b42-8ab9-78e5170eb674 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:43.567466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d91de3-8024-4b42-8ab9-78e5170eb674 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:43.567732 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d91de3-8024-4b42-8ab9-78e5170eb674 None admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:43.568442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d91de3-8024-4b42-8ab9-78e5170eb674 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:43.568741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d91de3-8024-4b42-8ab9-78e5170eb674 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:43.570154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d91de3-8024-4b42-8ab9-78e5170eb674 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:43.583291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 627/1259] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:08:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:08:43.589743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be15a02-6b7d-46c3-9033-c5f978fa4787 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:43.620146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:43.706276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be15a02-6b7d-46c3-9033-c5f978fa4787 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:43.741722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d11512-71a3-4d9b-81c9-e34d13eb5a1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:43.742319 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d11512-71a3-4d9b-81c9-e34d13eb5a1b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:43.742593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d11512-71a3-4d9b-81c9-e34d13eb5a1b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:43.742880 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d11512-71a3-4d9b-81c9-e34d13eb5a1b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:43.743576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d11512-71a3-4d9b-81c9-e34d13eb5a1b None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=972b0f1dbfef4d85a6dfc4c5fd839bde) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:43.743874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d11512-71a3-4d9b-81c9-e34d13eb5a1b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:43.745309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d11512-71a3-4d9b-81c9-e34d13eb5a1b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:43.751782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 633/1260] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:08:43 2022] GET /identity/v3/role_assignments?user.id=972b0f1dbfef4d85a6dfc4c5fd839bde&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:45.679413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28599ffc-5b61-40f3-9d41-4c40deca37fc None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:45.679919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28599ffc-5b61-40f3-9d41-4c40deca37fc None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:45.679919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28599ffc-5b61-40f3-9d41-4c40deca37fc None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:45.681247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 628/1261] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:08:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:08:45.687214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ff9313-f863-47ea-89e6-08745e9165ad None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:45.687659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ff9313-f863-47ea-89e6-08745e9165ad None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:45.687786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ff9313-f863-47ea-89e6-08745e9165ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:45.707790 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-d6ff9313-f863-47ea-89e6-08745e9165ad None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:08:45.709051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d6ff9313-f863-47ea-89e6-08745e9165ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:08:45.780825 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 634/1262] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:08:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:08:46.047918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28599ffc-5b61-40f3-9d41-4c40deca37fc None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:46.060036 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:46.155768 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28599ffc-5b61-40f3-9d41-4c40deca37fc None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:46.195788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586510f9-80b1-45cb-8a87-2fa911179712 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:46.196487 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586510f9-80b1-45cb-8a87-2fa911179712 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:46.196784 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586510f9-80b1-45cb-8a87-2fa911179712 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:46.197057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586510f9-80b1-45cb-8a87-2fa911179712 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:46.207714 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586510f9-80b1-45cb-8a87-2fa911179712 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:08:46.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-586510f9-80b1-45cb-8a87-2fa911179712 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:08:46.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:46.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:46.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:46.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:46.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:46.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:46.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:46.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:46.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:46.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:46.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:46.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:08:46.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:08:46.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:08:46.212446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:08:46.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:08:46.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:08:46.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:08:46.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:08:46.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:46.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:46.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:46.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:46.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:46.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:46.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:46.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:46.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:46.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:46.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:46.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:46.215742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:46.215742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:46.215742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:46.215742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:46.215742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:08:46.215742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:08:46.215742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:08:46.215742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:08:46.215742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:08:46.215742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:08:46.215742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:08:46.215742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:08:46.215742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 629/1263] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:08:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:46.221012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ff9313-f863-47ea-89e6-08745e9165ad None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:46.241081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:46.329792 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ff9313-f863-47ea-89e6-08745e9165ad None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:46.366389 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0728dd-f624-4de4-b929-2f5fe28ee910 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:46.366989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0728dd-f624-4de4-b929-2f5fe28ee910 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:46.367261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0728dd-f624-4de4-b929-2f5fe28ee910 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:46.367535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0728dd-f624-4de4-b929-2f5fe28ee910 None admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:46.368198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0728dd-f624-4de4-b929-2f5fe28ee910 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:46.368494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0728dd-f624-4de4-b929-2f5fe28ee910 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:46.369825 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0728dd-f624-4de4-b929-2f5fe28ee910 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:46.377997 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 635/1264] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:08:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:46.386434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586510f9-80b1-45cb-8a87-2fa911179712 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:46.459331 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:46.492243 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586510f9-80b1-45cb-8a87-2fa911179712 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:46.525894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d182d1-f30b-4097-afc9-ec912d09cf66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:46.526512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d182d1-f30b-4097-afc9-ec912d09cf66 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:46.526785 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d182d1-f30b-4097-afc9-ec912d09cf66 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:46.527051 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d182d1-f30b-4097-afc9-ec912d09cf66 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:46.538136 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d182d1-f30b-4097-afc9-ec912d09cf66 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:46.539564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d182d1-f30b-4097-afc9-ec912d09cf66 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:46.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-e8d182d1-f30b-4097-afc9-ec912d09cf66 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:46.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:46.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:46.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:46.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:46.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:46.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:46.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:46.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:46.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:46.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:46.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:46.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:08:46.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:08:46.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:08:46.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:46.559215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:08:46.560405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:08:46.560405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:46.560405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:08:46.560405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 630/1265] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 21 01:08:46.562939 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0728dd-f624-4de4-b929-2f5fe28ee910 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:46.645029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:46.665064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0728dd-f624-4de4-b929-2f5fe28ee910 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:46.702173 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685dc208-c429-430a-b472-12800818c15d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:46.702822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685dc208-c429-430a-b472-12800818c15d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:46.703049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685dc208-c429-430a-b472-12800818c15d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:46.703322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685dc208-c429-430a-b472-12800818c15d None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:46.703953 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685dc208-c429-430a-b472-12800818c15d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:46.704304 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685dc208-c429-430a-b472-12800818c15d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:46.705690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685dc208-c429-430a-b472-12800818c15d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:46.718681 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 636/1266] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:08:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:46.724763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d182d1-f30b-4097-afc9-ec912d09cf66 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:46.831045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d182d1-f30b-4097-afc9-ec912d09cf66 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:46.852199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:46.866318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4aeca68-e220-406d-9066-84e75af344c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:46.866951 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4aeca68-e220-406d-9066-84e75af344c8 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:46.867225 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4aeca68-e220-406d-9066-84e75af344c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:46.867491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4aeca68-e220-406d-9066-84e75af344c8 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:46.875779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4aeca68-e220-406d-9066-84e75af344c8 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:46.877255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4aeca68-e220-406d-9066-84e75af344c8 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:46.889094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-c4aeca68-e220-406d-9066-84e75af344c8 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 21 01:08:46.889094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:46.889094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:46.889094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:46.889094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:46.889094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:46.889094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:46.889094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:46.889094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:46.889094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:46.889094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:46.889094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:46.889094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:08:46.889094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:08:46.889094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:08:46.889094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:46.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:46.893496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:46.893496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:46.893496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:08:46.893496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:08:46.893496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:08:46.893496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:08:46.893496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:08:46.893496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:08:46.893496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 21 01:08:46.893496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:08:46.893496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 631/1267] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:46 2022] GET /identity/v3/users/placement => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:46.897416 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685dc208-c429-430a-b472-12800818c15d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:47.009256 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685dc208-c429-430a-b472-12800818c15d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:47.043411 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cb4ca1-07c6-49db-8ef8-c882df562d9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:47.044011 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cb4ca1-07c6-49db-8ef8-c882df562d9e None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:47.044316 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cb4ca1-07c6-49db-8ef8-c882df562d9e None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:47.044581 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cb4ca1-07c6-49db-8ef8-c882df562d9e None admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:47.045306 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cb4ca1-07c6-49db-8ef8-c882df562d9e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:47.045599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cb4ca1-07c6-49db-8ef8-c882df562d9e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:47.047010 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cb4ca1-07c6-49db-8ef8-c882df562d9e None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:47.056666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:47.065384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 637/1268] 10.222.0.47 () {56 vars in 1143 bytes} [Sun Aug 21 01:08:46 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:47.071186 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4aeca68-e220-406d-9066-84e75af344c8 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:47.182853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4aeca68-e220-406d-9066-84e75af344c8 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:47.226791 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336be9e0-39f4-47d1-8e3f-8f8da043a73d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:47.227432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336be9e0-39f4-47d1-8e3f-8f8da043a73d None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:47.227712 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336be9e0-39f4-47d1-8e3f-8f8da043a73d None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:47.227981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336be9e0-39f4-47d1-8e3f-8f8da043a73d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:47.240135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336be9e0-39f4-47d1-8e3f-8f8da043a73d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:47.241614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336be9e0-39f4-47d1-8e3f-8f8da043a73d None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:47.256973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-336be9e0-39f4-47d1-8e3f-8f8da043a73d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:47.256973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:47.256973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:47.256973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:47.256973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:47.256973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:47.256973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:47.256973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:47.256973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:47.256973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:47.256973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:47.256973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:47.256973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:08:47.256973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:08:47.256973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:08:47.256973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:47.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:08:47.259678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:08:47.259678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:47.259678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:08:47.259678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 632/1269] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:47.265418 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cb4ca1-07c6-49db-8ef8-c882df562d9e None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:47.375639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cb4ca1-07c6-49db-8ef8-c882df562d9e None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:47.410513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8deded6-2867-4aba-83e5-8106b8bafd8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:47.411123 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8deded6-2867-4aba-83e5-8106b8bafd8f None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:47.411403 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8deded6-2867-4aba-83e5-8106b8bafd8f None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:47.411675 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8deded6-2867-4aba-83e5-8106b8bafd8f None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:47.412448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8deded6-2867-4aba-83e5-8106b8bafd8f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:47.412826 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8deded6-2867-4aba-83e5-8106b8bafd8f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:47.414450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8deded6-2867-4aba-83e5-8106b8bafd8f None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:47.426646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 638/1270] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:08:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:47.432981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336be9e0-39f4-47d1-8e3f-8f8da043a73d None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:47.468546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:47.546179 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336be9e0-39f4-47d1-8e3f-8f8da043a73d None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:47.581064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43504bdb-277e-4444-9ce3-bbb9eaf45d95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:47.581700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43504bdb-277e-4444-9ce3-bbb9eaf45d95 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:47.581973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43504bdb-277e-4444-9ce3-bbb9eaf45d95 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:47.582242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43504bdb-277e-4444-9ce3-bbb9eaf45d95 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:47.589910 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43504bdb-277e-4444-9ce3-bbb9eaf45d95 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:47.592567 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43504bdb-277e-4444-9ce3-bbb9eaf45d95 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:47.604495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-43504bdb-277e-4444-9ce3-bbb9eaf45d95 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:08:47.604495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:47.604495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:47.604495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:47.604495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:47.604495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:47.604495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:47.604495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:47.604495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:47.604495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:47.604495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:47.604495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:47.604495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:08:47.604495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:08:47.604495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:47.605500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:47.607596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:08:47.607596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:08:47.607596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:08:47.607596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:08:47.607596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:08:47.607596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:08:47.607596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:08:47.607596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:08:47.607596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 633/1271] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:08:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:47.612849 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8deded6-2867-4aba-83e5-8106b8bafd8f None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:47.677108 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:47.721314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8deded6-2867-4aba-83e5-8106b8bafd8f None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:47.761648 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f002fc-a213-4e16-8a25-2067c457ab7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:47.762266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f002fc-a213-4e16-8a25-2067c457ab7c None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:47.762621 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f002fc-a213-4e16-8a25-2067c457ab7c None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:47.762961 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f002fc-a213-4e16-8a25-2067c457ab7c None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:47.763586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f002fc-a213-4e16-8a25-2067c457ab7c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:47.763888 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f002fc-a213-4e16-8a25-2067c457ab7c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:47.765371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f002fc-a213-4e16-8a25-2067c457ab7c None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:47.777523 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 639/1272] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:08:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:08:47.784546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43504bdb-277e-4444-9ce3-bbb9eaf45d95 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:47.883676 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:47.893890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43504bdb-277e-4444-9ce3-bbb9eaf45d95 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:47.942045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49c420a-5dad-4ff2-a774-a6ab59b4e503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:47.942723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c420a-5dad-4ff2-a774-a6ab59b4e503 None admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:47.943391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c420a-5dad-4ff2-a774-a6ab59b4e503 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:47.943391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c420a-5dad-4ff2-a774-a6ab59b4e503 None admin] PATH_INFO: `/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/972b0f1dbfef4d85a6dfc4c5fd839bde/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:47.945262 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49c420a-5dad-4ff2-a774-a6ab59b4e503 None admin] RBAC: Authorizing `identity:create_grant(project_id=75a3bf3f3a4244d18fde47d492657db4, user_id=972b0f1dbfef4d85a6dfc4c5fd839bde, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:47.946714 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49c420a-5dad-4ff2-a774-a6ab59b4e503 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:48.070702 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 634/1273] 10.222.0.47 () {58 vars in 1439 bytes} [Sun Aug 21 01:08:47 2022] PUT /identity/v3/projects/75a3bf3f3a4244d18fde47d492657db4/users/972b0f1dbfef4d85a6dfc4c5fd839bde/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:08:50.020660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3bfdd2-7ef3-48de-bb34-098bf635708b None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:50.021351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3bfdd2-7ef3-48de-bb34-098bf635708b None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:50.021351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3bfdd2-7ef3-48de-bb34-098bf635708b None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:50.022549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 640/1274] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:08:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:08:50.028839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4b5db0-faca-4ba3-a402-b2149db32158 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:50.029647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4b5db0-faca-4ba3-a402-b2149db32158 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:50.029806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4b5db0-faca-4ba3-a402-b2149db32158 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:50.057430 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-1f4b5db0-faca-4ba3-a402-b2149db32158 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:08:50.058766 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1f4b5db0-faca-4ba3-a402-b2149db32158 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:08:50.147525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 635/1275] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:08:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:08:50.419494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3bfdd2-7ef3-48de-bb34-098bf635708b None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:50.552056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3bfdd2-7ef3-48de-bb34-098bf635708b None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:50.570240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:50.587692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebc41c7-5230-407f-8381-24ce9cab3794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:50.588363 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebc41c7-5230-407f-8381-24ce9cab3794 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:50.588632 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebc41c7-5230-407f-8381-24ce9cab3794 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:50.588908 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebc41c7-5230-407f-8381-24ce9cab3794 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:50.600043 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebc41c7-5230-407f-8381-24ce9cab3794 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 21 01:08:50.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-5ebc41c7-5230-407f-8381-24ce9cab3794 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:08:50.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:50.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:50.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:50.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:50.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:50.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:50.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:50.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:50.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:50.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:50.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:50.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 21 01:08:50.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 21 01:08:50.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 21 01:08:50.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 21 01:08:50.606484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 21 01:08:50.606484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 21 01:08:50.606484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 21 01:08:50.606484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 21 01:08:50.606484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:50.606484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:50.606484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:50.606484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:50.606484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:50.606484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:50.606484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:50.606484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:50.606484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:50.606484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:50.606484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:50.606484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:50.608608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:50.608608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:50.608608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:50.608608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:50.608608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 21 01:08:50.608608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 21 01:08:50.608608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 21 01:08:50.608608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 21 01:08:50.608608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 21 01:08:50.608608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 21 01:08:50.608608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 21 01:08:50.608608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:50.608608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 641/1276] 10.222.0.47 () {56 vars in 1096 bytes} [Sun Aug 21 01:08:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:50.615492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4b5db0-faca-4ba3-a402-b2149db32158 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:50.718023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:50.741284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4b5db0-faca-4ba3-a402-b2149db32158 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:50.781110 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50388fbb-6c73-498f-9340-219271d755db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:50.782473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50388fbb-6c73-498f-9340-219271d755db None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:50.782985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50388fbb-6c73-498f-9340-219271d755db None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:50.783484 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50388fbb-6c73-498f-9340-219271d755db None admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:50.784496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50388fbb-6c73-498f-9340-219271d755db None admin] RBAC: Adding query filter params (name=admin) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:50.784977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50388fbb-6c73-498f-9340-219271d755db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:50.787163 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50388fbb-6c73-498f-9340-219271d755db None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:50.800592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 636/1277] 10.222.0.47 () {56 vars in 1099 bytes} [Sun Aug 21 01:08:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:50.809629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebc41c7-5230-407f-8381-24ce9cab3794 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:50.949795 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebc41c7-5230-407f-8381-24ce9cab3794 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:50.988691 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a100ef4-7b8d-40cc-a174-a4d38eaf7d61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:50.989350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a100ef4-7b8d-40cc-a174-a4d38eaf7d61 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:50.989720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a100ef4-7b8d-40cc-a174-a4d38eaf7d61 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:50.989993 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a100ef4-7b8d-40cc-a174-a4d38eaf7d61 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:51.002489 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a100ef4-7b8d-40cc-a174-a4d38eaf7d61 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:51.003961 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a100ef4-7b8d-40cc-a174-a4d38eaf7d61 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:51.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-3a100ef4-7b8d-40cc-a174-a4d38eaf7d61 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:51.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:51.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:51.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:51.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:51.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:51.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:51.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:51.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:51.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:51.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:51.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:51.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:08:51.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:08:51.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:08:51.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:51.022466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:08:51.023689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:08:51.023689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:51.023689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:51.023689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 642/1278] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:51.029056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50388fbb-6c73-498f-9340-219271d755db None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:51.154976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50388fbb-6c73-498f-9340-219271d755db None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:51.198860 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c163865-c6f0-4218-9b17-b1402e1c2a96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:51.199588 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c163865-c6f0-4218-9b17-b1402e1c2a96 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:51.200598 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c163865-c6f0-4218-9b17-b1402e1c2a96 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:51.200853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c163865-c6f0-4218-9b17-b1402e1c2a96 None admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:51.201625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c163865-c6f0-4218-9b17-b1402e1c2a96 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:51.201995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c163865-c6f0-4218-9b17-b1402e1c2a96 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:51.203699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c163865-c6f0-4218-9b17-b1402e1c2a96 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:51.219727 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 637/1279] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:08:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:51.227577 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a100ef4-7b8d-40cc-a174-a4d38eaf7d61 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:51.270647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:51.387831 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a100ef4-7b8d-40cc-a174-a4d38eaf7d61 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:51.437452 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3664b9-85e4-48c1-aed7-513f5ce6238d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:51.438187 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3664b9-85e4-48c1-aed7-513f5ce6238d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:51.438506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3664b9-85e4-48c1-aed7-513f5ce6238d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:51.438845 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3664b9-85e4-48c1-aed7-513f5ce6238d None admin] PATH_INFO: `/v3/users/placement` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:51.447397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3664b9-85e4-48c1-aed7-513f5ce6238d None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:51.448922 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3664b9-85e4-48c1-aed7-513f5ce6238d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:51.463459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-7e3664b9-85e4-48c1-aed7-513f5ce6238d None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 21 01:08:51.463459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:51.463459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:51.463459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:51.463459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:51.463459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:51.463459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:51.463459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:51.463459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:51.463459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:51.463459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:51.463459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:51.463459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 21 01:08:51.463459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 21 01:08:51.463459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 21 01:08:51.463459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:51.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:51.466892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:51.466892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:51.466892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 21 01:08:51.466892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 21 01:08:51.466892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 21 01:08:51.466892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 21 01:08:51.466892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 21 01:08:51.466892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 21 01:08:51.466892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 21 01:08:51.466892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:51.466892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 643/1280] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:51 2022] GET /identity/v3/users/placement => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:51.472202 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c163865-c6f0-4218-9b17-b1402e1c2a96 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:51.476465 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:51.599886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c163865-c6f0-4218-9b17-b1402e1c2a96 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:51.656420 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6900ca-d0e0-4b3d-bcf1-1e87ecc5385a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:51.657055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6900ca-d0e0-4b3d-bcf1-1e87ecc5385a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:51.657333 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6900ca-d0e0-4b3d-bcf1-1e87ecc5385a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:51.657601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6900ca-d0e0-4b3d-bcf1-1e87ecc5385a None admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:51.658370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6900ca-d0e0-4b3d-bcf1-1e87ecc5385a None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:51.658684 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6900ca-d0e0-4b3d-bcf1-1e87ecc5385a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:51.660195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6900ca-d0e0-4b3d-bcf1-1e87ecc5385a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:51.682975 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 638/1281] 10.222.0.47 () {56 vars in 1143 bytes} [Sun Aug 21 01:08:51 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:51.690313 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3664b9-85e4-48c1-aed7-513f5ce6238d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:51.816074 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:51.826856 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3664b9-85e4-48c1-aed7-513f5ce6238d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:51.862091 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f402073-5b83-42fc-838d-9e0547d8eb57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:51.862716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f402073-5b83-42fc-838d-9e0547d8eb57 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:51.862990 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f402073-5b83-42fc-838d-9e0547d8eb57 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:51.863257 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f402073-5b83-42fc-838d-9e0547d8eb57 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:51.936068 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f402073-5b83-42fc-838d-9e0547d8eb57 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:51.937608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f402073-5b83-42fc-838d-9e0547d8eb57 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:52.031421 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:52.042309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 644/1282] 10.222.0.47 () {58 vars in 1282 bytes} [Sun Aug 21 01:08:51 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 21 01:08:52.048672 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6900ca-d0e0-4b3d-bcf1-1e87ecc5385a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:52.121058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:52.186629 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6900ca-d0e0-4b3d-bcf1-1e87ecc5385a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:52.231737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8af529-eea2-4ad9-8af2-afe49c428c6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:52.232499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8af529-eea2-4ad9-8af2-afe49c428c6a None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:52.232766 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8af529-eea2-4ad9-8af2-afe49c428c6a None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:52.233059 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8af529-eea2-4ad9-8af2-afe49c428c6a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:52.244241 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8af529-eea2-4ad9-8af2-afe49c428c6a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:52.245707 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8af529-eea2-4ad9-8af2-afe49c428c6a None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:52.262539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-4b8af529-eea2-4ad9-8af2-afe49c428c6a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:52.262539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:52.262539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:52.262539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:52.262539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:52.262539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:52.262539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:52.262539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:52.262539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:52.262539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:52.262539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:52.262539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:52.262539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 21 01:08:52.262539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 21 01:08:52.262539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 21 01:08:52.262539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:52.263728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 21 01:08:52.264907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 21 01:08:52.264907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 21 01:08:52.264907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:08:52.266850 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 639/1283] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:08:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:52.272130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f402073-5b83-42fc-838d-9e0547d8eb57 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:52.382217 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f402073-5b83-42fc-838d-9e0547d8eb57 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:52.419967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf1a9b7-9caf-4e4f-85bb-a0c5f2c8777f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:52.420686 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf1a9b7-9caf-4e4f-85bb-a0c5f2c8777f None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:52.420962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf1a9b7-9caf-4e4f-85bb-a0c5f2c8777f None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:52.421232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf1a9b7-9caf-4e4f-85bb-a0c5f2c8777f None admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:52.421882 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf1a9b7-9caf-4e4f-85bb-a0c5f2c8777f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:52.422179 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf1a9b7-9caf-4e4f-85bb-a0c5f2c8777f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:52.423533 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf1a9b7-9caf-4e4f-85bb-a0c5f2c8777f None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:52.436085 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 645/1284] 10.222.0.47 () {56 vars in 1109 bytes} [Sun Aug 21 01:08:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:08:52.442214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8af529-eea2-4ad9-8af2-afe49c428c6a None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:52.556597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8af529-eea2-4ad9-8af2-afe49c428c6a None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:52.575296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:52.595956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef03c84-9b1d-4fe3-9234-5a4b8619b75c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:52.596614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef03c84-9b1d-4fe3-9234-5a4b8619b75c None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:52.596894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef03c84-9b1d-4fe3-9234-5a4b8619b75c None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:52.597160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef03c84-9b1d-4fe3-9234-5a4b8619b75c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:52.604317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef03c84-9b1d-4fe3-9234-5a4b8619b75c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:52.606298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef03c84-9b1d-4fe3-9234-5a4b8619b75c None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:52.632849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application [None req-6ef03c84-9b1d-4fe3-9234-5a4b8619b75c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:08:52.632849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:52.632849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:52.632849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:52.632849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:52.632849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:52.632849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:52.632849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:52.632849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:52.632849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:52.632849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:52.632849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:52.632849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 21 01:08:52.632849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 21 01:08:52.632849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:52.634638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:52.636470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 21 01:08:52.636470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 21 01:08:52.636470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 21 01:08:52.636470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 21 01:08:52.636470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 21 01:08:52.636470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 21 01:08:52.636470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 21 01:08:52.636470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: ERROR keystone.server.flask.application Aug 21 01:08:52.636470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 640/1285] 10.222.0.47 () {56 vars in 1111 bytes} [Sun Aug 21 01:08:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:52.640611 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf1a9b7-9caf-4e4f-85bb-a0c5f2c8777f None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:52.698084 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:52.758126 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf1a9b7-9caf-4e4f-85bb-a0c5f2c8777f None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:52.795710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c3eea4-a117-4cd0-aaca-4464c7a393df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:52.796356 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c3eea4-a117-4cd0-aaca-4464c7a393df None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:52.797153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c3eea4-a117-4cd0-aaca-4464c7a393df None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:52.797437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c3eea4-a117-4cd0-aaca-4464c7a393df None admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:52.798127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c3eea4-a117-4cd0-aaca-4464c7a393df None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:52.798431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c3eea4-a117-4cd0-aaca-4464c7a393df None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:52.799917 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c3eea4-a117-4cd0-aaca-4464c7a393df None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:52.813841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 646/1286] 10.222.0.47 () {56 vars in 1148 bytes} [Sun Aug 21 01:08:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:08:52.820201 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef03c84-9b1d-4fe3-9234-5a4b8619b75c None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:52.923940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef03c84-9b1d-4fe3-9234-5a4b8619b75c None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:52.962178 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8c4481-c319-4f38-9d00-fcb1efcd5966 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:52.962767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8c4481-c319-4f38-9d00-fcb1efcd5966 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:52.963038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8c4481-c319-4f38-9d00-fcb1efcd5966 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:52.963302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8c4481-c319-4f38-9d00-fcb1efcd5966 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:52.964008 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8c4481-c319-4f38-9d00-fcb1efcd5966 None admin] RBAC: Adding query filter params (role.id=ea4472fe9a96464888f3eb593eb46674, scope.project.id=75a3bf3f3a4244d18fde47d492657db4, user.id=972b0f1dbfef4d85a6dfc4c5fd839bde) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:52.964405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8c4481-c319-4f38-9d00-fcb1efcd5966 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:52.966358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8c4481-c319-4f38-9d00-fcb1efcd5966 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:52.972128 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 641/1287] 10.222.0.47 () {56 vars in 1374 bytes} [Sun Aug 21 01:08:52 2022] GET /identity/v3/role_assignments?user.id=972b0f1dbfef4d85a6dfc4c5fd839bde&scope.project.id=75a3bf3f3a4244d18fde47d492657db4&role.id=ea4472fe9a96464888f3eb593eb46674 => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:54.937309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f899c46d-c4b5-44eb-818a-78c90273b3dd None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:54.937960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f899c46d-c4b5-44eb-818a-78c90273b3dd None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:54.937960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f899c46d-c4b5-44eb-818a-78c90273b3dd None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:54.939255 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 647/1288] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:08:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:08:54.945167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35656dc-c3da-476f-97cd-4f094f882c1a None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:54.945694 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35656dc-c3da-476f-97cd-4f094f882c1a None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:54.945694 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35656dc-c3da-476f-97cd-4f094f882c1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:54.969355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-e35656dc-c3da-476f-97cd-4f094f882c1a None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:08:54.970539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e35656dc-c3da-476f-97cd-4f094f882c1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:08:55.002171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:55.046349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 642/1289] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:08:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:08:55.316916 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f899c46d-c4b5-44eb-818a-78c90273b3dd None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:55.453846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f899c46d-c4b5-44eb-818a-78c90273b3dd None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:55.466535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:55.492905 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e380e2-f76a-4ed5-8b6f-2ff711d24aa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:55.493586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e380e2-f76a-4ed5-8b6f-2ff711d24aa4 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:55.493859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e380e2-f76a-4ed5-8b6f-2ff711d24aa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:55.494126 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e380e2-f76a-4ed5-8b6f-2ff711d24aa4 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:55.494704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e380e2-f76a-4ed5-8b6f-2ff711d24aa4 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:55.496701 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e380e2-f76a-4ed5-8b6f-2ff711d24aa4 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-b6e380e2-f76a-4ed5-8b6f-2ff711d24aa4 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:08:55.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:08:55.508639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:08:55.508639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:08:55.508639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:08:55.508639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:08:55.508639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:08:55.508639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:08:55.508639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:08:55.508639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:08:55.508639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:08:55.508639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:08:55.508639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:08:55.508639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:08:55.508639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:08:55.508639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:08:55.508639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:08:55.508639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:08:55.510576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:08:55.510576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:08:55.510576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:08:55.510576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:08:55.510576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:08:55.510576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:08:55.510576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:08:55.510576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:08:55.510576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:08:55.510576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:08:55.510576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:08:55.510576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:08:55.510576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 21 01:08:55.510576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:08:55.510576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 648/1290] 10.222.0.47 () {56 vars in 1117 bytes} [Sun Aug 21 01:08:55 2022] GET /identity/v3/services/placement => generated 90 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:08:55.515387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35656dc-c3da-476f-97cd-4f094f882c1a None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:55.635887 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35656dc-c3da-476f-97cd-4f094f882c1a None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:55.677545 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199b4554-aefc-485d-a669-22f129a1ecb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:55.678426 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199b4554-aefc-485d-a669-22f129a1ecb5 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:55.678706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199b4554-aefc-485d-a669-22f129a1ecb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:55.678974 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199b4554-aefc-485d-a669-22f129a1ecb5 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:55.679747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199b4554-aefc-485d-a669-22f129a1ecb5 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:55.679913 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199b4554-aefc-485d-a669-22f129a1ecb5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:55.681472 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199b4554-aefc-485d-a669-22f129a1ecb5 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:55.689629 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 643/1291] 10.222.0.47 () {56 vars in 1116 bytes} [Sun Aug 21 01:08:55 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:08:55.695780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e380e2-f76a-4ed5-8b6f-2ff711d24aa4 None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:55.812273 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e380e2-f76a-4ed5-8b6f-2ff711d24aa4 None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:55.854569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cdbc1b-7468-48f0-b6ca-968b0d60899b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:55.855204 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cdbc1b-7468-48f0-b6ca-968b0d60899b None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:55.856034 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cdbc1b-7468-48f0-b6ca-968b0d60899b None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:55.856331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cdbc1b-7468-48f0-b6ca-968b0d60899b None admin] PATH_INFO: `/v3/services` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:55.856957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cdbc1b-7468-48f0-b6ca-968b0d60899b None admin] RBAC: Adding query filter params (type=placement) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:08:55.857253 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cdbc1b-7468-48f0-b6ca-968b0d60899b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:55.858601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cdbc1b-7468-48f0-b6ca-968b0d60899b None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:55.863456 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 649/1292] 10.222.0.47 () {56 vars in 1116 bytes} [Sun Aug 21 01:08:55 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:08:57.799052 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69bd44c-70ec-44f7-a853-670b348668e6 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:57.799634 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69bd44c-70ec-44f7-a853-670b348668e6 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:57.799895 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69bd44c-70ec-44f7-a853-670b348668e6 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:57.802940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 644/1293] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:08:57 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:08:57.812056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b112eb27-19b8-4d8f-8447-3a6ad56aec73 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:57.812709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b112eb27-19b8-4d8f-8447-3a6ad56aec73 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:57.813035 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b112eb27-19b8-4d8f-8447-3a6ad56aec73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:57.845166 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-b112eb27-19b8-4d8f-8447-3a6ad56aec73 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:08:57.847092 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b112eb27-19b8-4d8f-8447-3a6ad56aec73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:08:57.899030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:57.960691 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 650/1294] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:08:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:08:58.250868 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69bd44c-70ec-44f7-a853-670b348668e6 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:08:58.297324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:08:58.386744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69bd44c-70ec-44f7-a853-670b348668e6 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:08:58.424680 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db31fc42-b28b-46dd-8164-1749967facac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:08:58.425294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db31fc42-b28b-46dd-8164-1749967facac None admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:08:58.425568 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db31fc42-b28b-46dd-8164-1749967facac None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:08:58.425837 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db31fc42-b28b-46dd-8164-1749967facac None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:08:58.426480 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db31fc42-b28b-46dd-8164-1749967facac None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:08:58.428093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db31fc42-b28b-46dd-8164-1749967facac None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:08:58.495311 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 645/1295] 10.222.0.47 () {60 vars in 1141 bytes} [Sun Aug 21 01:08:58 2022] POST /identity/v3/services => generated 246 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:09:00.434806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd705173-d8b0-45fd-b171-38e16e05ce4a None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:09:00.435343 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd705173-d8b0-45fd-b171-38e16e05ce4a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:09:00.435463 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd705173-d8b0-45fd-b171-38e16e05ce4a None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:09:00.437077 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 651/1296] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:09:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:09:00.449348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084af3de-70cd-4636-b3d6-33c6b6df333c None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:09:00.449899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084af3de-70cd-4636-b3d6-33c6b6df333c None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:09:00.449899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084af3de-70cd-4636-b3d6-33c6b6df333c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:09:00.470394 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-084af3de-70cd-4636-b3d6-33c6b6df333c None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:09:00.471678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-084af3de-70cd-4636-b3d6-33c6b6df333c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:09:00.552234 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 646/1297] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:09:00 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:09:00.836086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd705173-d8b0-45fd-b171-38e16e05ce4a None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:09:00.943120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd705173-d8b0-45fd-b171-38e16e05ce4a None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:09:00.976650 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6f0f3b-ba8a-4828-aa7d-ffc5480c350d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:09:00.977272 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6f0f3b-ba8a-4828-aa7d-ffc5480c350d None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:09:00.977544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6f0f3b-ba8a-4828-aa7d-ffc5480c350d None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:09:00.977806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6f0f3b-ba8a-4828-aa7d-ffc5480c350d None admin] PATH_INFO: `/v3/services/placement` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:09:00.978452 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6f0f3b-ba8a-4828-aa7d-ffc5480c350d None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:09:00.980392 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6f0f3b-ba8a-4828-aa7d-ffc5480c350d None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-fb6f0f3b-ba8a-4828-aa7d-ffc5480c350d None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:09:00.989150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:09:00.990364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:09:00.990364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:09:00.990364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:09:00.990364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:09:00.990364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:09:00.990364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:09:00.990364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:09:00.990364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:09:00.990364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:09:00.990364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:09:00.990364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:09:00.990364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:09:00.990364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:09:00.990364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:09:00.990364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:09:00.990364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:09:00.991619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:09:00.991619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:09:00.991619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:09:00.991619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:09:00.991619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:09:00.991619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:09:00.991619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:09:00.991619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:09:00.991619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:09:00.991619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:09:00.991619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:09:00.991619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:09:00.991619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 21 01:09:00.991619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:09:00.991619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 652/1298] 10.222.0.47 () {56 vars in 1117 bytes} [Sun Aug 21 01:09:00 2022] GET /identity/v3/services/placement => generated 90 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:09:00.996930 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084af3de-70cd-4636-b3d6-33c6b6df333c None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:09:01.016145 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:09:01.097691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084af3de-70cd-4636-b3d6-33c6b6df333c None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:09:01.133255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c23a7de-43eb-44c0-b022-aaab0b17f064 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:09:01.133865 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c23a7de-43eb-44c0-b022-aaab0b17f064 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:09:01.134138 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c23a7de-43eb-44c0-b022-aaab0b17f064 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:09:01.134404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c23a7de-43eb-44c0-b022-aaab0b17f064 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:09:01.135029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c23a7de-43eb-44c0-b022-aaab0b17f064 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:09:01.135324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c23a7de-43eb-44c0-b022-aaab0b17f064 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:09:01.136720 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c23a7de-43eb-44c0-b022-aaab0b17f064 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:09:01.141895 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 647/1299] 10.222.0.47 () {56 vars in 1116 bytes} [Sun Aug 21 01:09:00 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:09:01.148335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6f0f3b-ba8a-4828-aa7d-ffc5480c350d None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:09:01.182242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:09:01.253846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6f0f3b-ba8a-4828-aa7d-ffc5480c350d None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:09:01.291011 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59702f62-10f2-4950-9ad9-760b8f0641b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:09:01.291615 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59702f62-10f2-4950-9ad9-760b8f0641b9 None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:09:01.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59702f62-10f2-4950-9ad9-760b8f0641b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:09:01.292873 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59702f62-10f2-4950-9ad9-760b8f0641b9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:09:01.293577 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59702f62-10f2-4950-9ad9-760b8f0641b9 None admin] RBAC: Adding query filter params (interface=public, service_id=a0ec150daf864bc4ac961c8491147b63, region_id=RegionOne) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:09:01.293884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59702f62-10f2-4950-9ad9-760b8f0641b9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:09:01.295275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59702f62-10f2-4950-9ad9-760b8f0641b9 None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:09:01.301261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 653/1300] 10.222.0.47 () {56 vars in 1251 bytes} [Sun Aug 21 01:09:01 2022] GET /identity/v3/endpoints?service_id=a0ec150daf864bc4ac961c8491147b63&interface=public®ion_id=RegionOne => generated 194 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:09:01.307627 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c23a7de-43eb-44c0-b022-aaab0b17f064 None admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:09:01.395769 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:09:01.419216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c23a7de-43eb-44c0-b022-aaab0b17f064 None admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:09:01.460626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bbecbe-a1ce-4636-9ebd-1ef23370ccd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:09:01.461278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bbecbe-a1ce-4636-9ebd-1ef23370ccd9 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:09:01.461492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bbecbe-a1ce-4636-9ebd-1ef23370ccd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:09:01.461826 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bbecbe-a1ce-4636-9ebd-1ef23370ccd9 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:09:01.462368 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bbecbe-a1ce-4636-9ebd-1ef23370ccd9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:09:01.463722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bbecbe-a1ce-4636-9ebd-1ef23370ccd9 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:09:01.471298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 648/1301] 10.222.0.47 () {56 vars in 1087 bytes} [Sun Aug 21 01:09:01 2022] GET /identity/v3/services => generated 2191 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:09:03.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30e1099-15ce-4318-afa4-eeb2bc2c9589 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:09:03.379484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30e1099-15ce-4318-afa4-eeb2bc2c9589 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:09:03.379484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30e1099-15ce-4318-afa4-eeb2bc2c9589 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:09:03.380867 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 654/1302] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:09:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:09:03.386202 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef61098b-d2ff-4909-ad8a-c38fe0a941ee None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:09:03.386610 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef61098b-d2ff-4909-ad8a-c38fe0a941ee None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:09:03.386740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef61098b-d2ff-4909-ad8a-c38fe0a941ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:09:03.404963 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-ef61098b-d2ff-4909-ad8a-c38fe0a941ee None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:09:03.406427 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ef61098b-d2ff-4909-ad8a-c38fe0a941ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:09:03.472980 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 649/1303] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:09:03 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 21 01:09:03.735330 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30e1099-15ce-4318-afa4-eeb2bc2c9589 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:09:03.831355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:09:03.837908 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30e1099-15ce-4318-afa4-eeb2bc2c9589 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:09:03.871698 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0668c5-611b-427b-bbbe-a2c96d01e3fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:09:03.872349 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0668c5-611b-427b-bbbe-a2c96d01e3fa None admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:09:03.872627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0668c5-611b-427b-bbbe-a2c96d01e3fa None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:09:03.872894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0668c5-611b-427b-bbbe-a2c96d01e3fa None admin] PATH_INFO: `/v3/services/placement` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:09:03.873461 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0668c5-611b-427b-bbbe-a2c96d01e3fa None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:09:03.874817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0668c5-611b-427b-bbbe-a2c96d01e3fa None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application [None req-fc0668c5-611b-427b-bbbe-a2c96d01e3fa None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 21 01:09:03.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 21 01:09:03.889660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 21 01:09:03.889660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 21 01:09:03.889660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 21 01:09:03.889660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 21 01:09:03.889660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 21 01:09:03.889660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 21 01:09:03.889660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 21 01:09:03.889660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 21 01:09:03.889660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 21 01:09:03.889660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 21 01:09:03.889660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 21 01:09:03.889660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 21 01:09:03.889660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 21 01:09:03.889660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 21 01:09:03.889660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 21 01:09:03.889660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.get_or_create( Aug 21 01:09:03.890730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 21 01:09:03.890730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application with Lock( Aug 21 01:09:03.890730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 21 01:09:03.890730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self._enter() Aug 21 01:09:03.890730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 21 01:09:03.890730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 21 01:09:03.890730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 21 01:09:03.890730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application return self.creator() Aug 21 01:09:03.890730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 21 01:09:03.890730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application created_value = creator( Aug 21 01:09:03.890730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 21 01:09:03.890730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 21 01:09:03.890730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 21 01:09:03.890730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: ERROR keystone.server.flask.application Aug 21 01:09:03.890730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 655/1304] 10.222.0.47 () {56 vars in 1117 bytes} [Sun Aug 21 01:09:03 2022] GET /identity/v3/services/placement => generated 90 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 21 01:09:03.892365 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef61098b-d2ff-4909-ad8a-c38fe0a941ee None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:09:03.974175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:09:03.997505 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef61098b-d2ff-4909-ad8a-c38fe0a941ee None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:09:04.034729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d257797-e939-4ca3-a4da-0aa06de998c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:09:04.035379 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d257797-e939-4ca3-a4da-0aa06de998c6 None admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:09:04.035651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d257797-e939-4ca3-a4da-0aa06de998c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:09:04.035917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d257797-e939-4ca3-a4da-0aa06de998c6 None admin] PATH_INFO: `/v3/services` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:09:04.036568 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d257797-e939-4ca3-a4da-0aa06de998c6 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:09:04.036865 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d257797-e939-4ca3-a4da-0aa06de998c6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:09:04.038226 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d257797-e939-4ca3-a4da-0aa06de998c6 None admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:09:04.043741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 650/1305] 10.222.0.47 () {56 vars in 1116 bytes} [Sun Aug 21 01:09:03 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:09:04.049701 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0668c5-611b-427b-bbbe-a2c96d01e3fa None admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:09:04.162828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0668c5-611b-427b-bbbe-a2c96d01e3fa None admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:09:04.197171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed07abf9-1132-48a3-8f44-fb99ef1077ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:09:04.197772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed07abf9-1132-48a3-8f44-fb99ef1077ea None admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:09:04.198655 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed07abf9-1132-48a3-8f44-fb99ef1077ea None admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:09:04.198941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed07abf9-1132-48a3-8f44-fb99ef1077ea None admin] PATH_INFO: `/v3/endpoints` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:09:04.199580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed07abf9-1132-48a3-8f44-fb99ef1077ea None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:09:04.200951 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed07abf9-1132-48a3-8f44-fb99ef1077ea None admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:09:04.318337 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 656/1306] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:09:04 2022] POST /identity/v3/endpoints => generated 331 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:09:14.049634 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=30 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:09:14.313448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:31.134095 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8e86d9-81a6-4fe0-90cf-a6fa30a7ef43 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:31.134864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8e86d9-81a6-4fe0-90cf-a6fa30a7ef43 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:31.134864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8e86d9-81a6-4fe0-90cf-a6fa30a7ef43 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:31.142192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335fe356-3518-4763-a86a-28e9e10b334b None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:31.142852 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335fe356-3518-4763-a86a-28e9e10b334b None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:31.142852 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335fe356-3518-4763-a86a-28e9e10b334b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:31.147317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 651/1307] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:16:31 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:16:31.236693 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-335fe356-3518-4763-a86a-28e9e10b334b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:16:31.237850 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-335fe356-3518-4763-a86a-28e9e10b334b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:31.376533 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 657/1308] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:16:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:16:31.681730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7401fe2-7d8d-47d9-83fc-59014a34ae46 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:31.682292 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7401fe2-7d8d-47d9-83fc-59014a34ae46 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:31.682711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7401fe2-7d8d-47d9-83fc-59014a34ae46 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:31.684689 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 652/1309] 10.222.0.47 () {54 vars in 965 bytes} [Sun Aug 21 01:16:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:16:31.694563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab41cc6-6fc4-4426-bdda-269bbcf87892 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:31.695247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab41cc6-6fc4-4426-bdda-269bbcf87892 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:31.695875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab41cc6-6fc4-4426-bdda-269bbcf87892 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:31.814267 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-6ab41cc6-6fc4-4426-bdda-269bbcf87892 None None] MFA Rules not processed for user `b532ddab1f5e467989edaa9e4b74bd72`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:16:31.815687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-6ab41cc6-6fc4-4426-bdda-269bbcf87892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:31.848934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:31.924366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 658/1310] 10.222.0.47 () {58 vars in 1062 bytes} [Sun Aug 21 01:16:31 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:16:31.934265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7401fe2-7d8d-47d9-83fc-59014a34ae46 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:16:31.936039 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-b7401fe2-7d8d-47d9-83fc-59014a34ae46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:32.053723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7401fe2-7d8d-47d9-83fc-59014a34ae46 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:16:32.141475 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6114f38d-039d-4646-9a5c-dbdb0ae2a4c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:16:32.142154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6114f38d-039d-4646-9a5c-dbdb0ae2a4c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:32.142452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6114f38d-039d-4646-9a5c-dbdb0ae2a4c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:32.142632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6114f38d-039d-4646-9a5c-dbdb0ae2a4c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:32.144296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6114f38d-039d-4646-9a5c-dbdb0ae2a4c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:32.207079 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:32.269330 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6114f38d-039d-4646-9a5c-dbdb0ae2a4c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:16:32.270940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6114f38d-039d-4646-9a5c-dbdb0ae2a4c3 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:16:32.291757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 653/1311] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:16:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:16:34.594092 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dc9cd1-1330-4fdb-833b-ded81b4b5f2d None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:34.594637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dc9cd1-1330-4fdb-833b-ded81b4b5f2d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:34.594637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dc9cd1-1330-4fdb-833b-ded81b4b5f2d None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:34.596011 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 659/1312] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:16:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:16:34.602002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf9cb6f-c161-46d1-9e97-7c4e3ec311ac None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:34.602522 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf9cb6f-c161-46d1-9e97-7c4e3ec311ac None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:34.602522 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf9cb6f-c161-46d1-9e97-7c4e3ec311ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:34.685853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-1bf9cb6f-c161-46d1-9e97-7c4e3ec311ac None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:16:34.687581 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1bf9cb6f-c161-46d1-9e97-7c4e3ec311ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:34.804932 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 654/1313] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:16:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:16:35.092493 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2dc9cd1-1330-4fdb-833b-ded81b4b5f2d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:16:35.094067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b2dc9cd1-1330-4fdb-833b-ded81b4b5f2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:35.209107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2dc9cd1-1330-4fdb-833b-ded81b4b5f2d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:16:35.300705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:35.303887 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc36fac-e25e-4d46-9922-e13628cfe532 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:16:35.304558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc36fac-e25e-4d46-9922-e13628cfe532 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:35.304833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc36fac-e25e-4d46-9922-e13628cfe532 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:35.305102 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc36fac-e25e-4d46-9922-e13628cfe532 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:35.306835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-efc36fac-e25e-4d46-9922-e13628cfe532 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:35.425692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc36fac-e25e-4d46-9922-e13628cfe532 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:16:35.427122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc36fac-e25e-4d46-9922-e13628cfe532 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:16:35.452184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 660/1314] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:16:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:16:38.061907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf4a11e-63b8-4be3-ba5a-855db069cd35 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:38.063066 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf4a11e-63b8-4be3-ba5a-855db069cd35 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:38.063066 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf4a11e-63b8-4be3-ba5a-855db069cd35 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:38.064696 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 655/1315] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:16:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:16:38.070814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3463f36c-d443-4e64-b3ea-abbf1ea58ff9 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:38.071355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3463f36c-d443-4e64-b3ea-abbf1ea58ff9 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:38.071488 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3463f36c-d443-4e64-b3ea-abbf1ea58ff9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:38.142097 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-3463f36c-d443-4e64-b3ea-abbf1ea58ff9 None None] MFA Rules not processed for user `4e36c8bcb5234cd29a8d01bb6f6d6d38`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:16:38.143351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-3463f36c-d443-4e64-b3ea-abbf1ea58ff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:38.200154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:38.365832 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 661/1316] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:16:38 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 297 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:16:38.671344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec1b7f8-3154-4264-aa35-2913449bc4a4 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:38.671617 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec1b7f8-3154-4264-aa35-2913449bc4a4 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:38.671880 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec1b7f8-3154-4264-aa35-2913449bc4a4 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:38.673261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 656/1317] 10.222.0.47 () {54 vars in 965 bytes} [Sun Aug 21 01:16:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:16:38.679856 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47b0a6d-6aa3-42d2-b8df-475d6a081696 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:38.680160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47b0a6d-6aa3-42d2-b8df-475d6a081696 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:38.680432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47b0a6d-6aa3-42d2-b8df-475d6a081696 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:38.784374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-b47b0a6d-6aa3-42d2-b8df-475d6a081696 None None] MFA Rules not processed for user `b532ddab1f5e467989edaa9e4b74bd72`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:16:38.786332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b47b0a6d-6aa3-42d2-b8df-475d6a081696 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:38.871443 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 662/1318] 10.222.0.47 () {58 vars in 1062 bytes} [Sun Aug 21 01:16:38 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:16:38.880660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec1b7f8-3154-4264-aa35-2913449bc4a4 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:16:38.882220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3ec1b7f8-3154-4264-aa35-2913449bc4a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:38.892898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:39.006216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec1b7f8-3154-4264-aa35-2913449bc4a4 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:16:39.084808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea962b14-5b31-46f0-b2f9-8d52cd2b9798 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:16:39.085517 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea962b14-5b31-46f0-b2f9-8d52cd2b9798 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:39.085664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea962b14-5b31-46f0-b2f9-8d52cd2b9798 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:39.085927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea962b14-5b31-46f0-b2f9-8d52cd2b9798 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:39.087541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ea962b14-5b31-46f0-b2f9-8d52cd2b9798 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:39.150132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:39.202245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea962b14-5b31-46f0-b2f9-8d52cd2b9798 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:16:39.203708 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea962b14-5b31-46f0-b2f9-8d52cd2b9798 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:16:39.223740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 657/1319] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:16:38 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:16:42.343325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb62cf0e-fd7f-4271-a3f6-bdf681cfe544 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:42.344151 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb62cf0e-fd7f-4271-a3f6-bdf681cfe544 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:42.344151 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb62cf0e-fd7f-4271-a3f6-bdf681cfe544 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:42.345263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 663/1320] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:16:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:16:42.350843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c00616-6519-491f-a456-efc922825314 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:42.351227 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c00616-6519-491f-a456-efc922825314 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:42.351399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c00616-6519-491f-a456-efc922825314 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:42.421351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-69c00616-6519-491f-a456-efc922825314 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:16:42.422708 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-69c00616-6519-491f-a456-efc922825314 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:42.522527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 658/1321] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:16:42 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:16:42.815921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb62cf0e-fd7f-4271-a3f6-bdf681cfe544 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:16:42.817494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-fb62cf0e-fd7f-4271-a3f6-bdf681cfe544 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:42.868935 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:42.923514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb62cf0e-fd7f-4271-a3f6-bdf681cfe544 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:16:43.001697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220b8ea0-ea66-4bdc-a7bc-3f0d98a0222a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:16:43.004656 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220b8ea0-ea66-4bdc-a7bc-3f0d98a0222a service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:43.007101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220b8ea0-ea66-4bdc-a7bc-3f0d98a0222a service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:43.007377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220b8ea0-ea66-4bdc-a7bc-3f0d98a0222a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:43.011352 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-220b8ea0-ea66-4bdc-a7bc-3f0d98a0222a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:43.109394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:43.127076 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220b8ea0-ea66-4bdc-a7bc-3f0d98a0222a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:16:43.128549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220b8ea0-ea66-4bdc-a7bc-3f0d98a0222a service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:16:43.147129 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 664/1322] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:16:42 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:16:46.205031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef53578f-944b-419e-96ed-ee88c0593642 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:46.205628 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef53578f-944b-419e-96ed-ee88c0593642 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:46.205628 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef53578f-944b-419e-96ed-ee88c0593642 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:46.206908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 659/1323] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:16:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:16:46.212700 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c8ee8c-85d8-48c3-8b0a-a059f072bc31 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:46.213398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c8ee8c-85d8-48c3-8b0a-a059f072bc31 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:46.213398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c8ee8c-85d8-48c3-8b0a-a059f072bc31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:46.297851 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-66c8ee8c-85d8-48c3-8b0a-a059f072bc31 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:16:46.299719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-66c8ee8c-85d8-48c3-8b0a-a059f072bc31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:46.428087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 665/1324] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:16:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:16:46.718651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef53578f-944b-419e-96ed-ee88c0593642 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:16:46.720240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ef53578f-944b-419e-96ed-ee88c0593642 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:46.741370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:46.849438 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef53578f-944b-419e-96ed-ee88c0593642 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:16:46.933389 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36260f22-1889-4d44-a918-f8a0ccbe1326 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:16:46.934189 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36260f22-1889-4d44-a918-f8a0ccbe1326 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:46.934789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36260f22-1889-4d44-a918-f8a0ccbe1326 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:46.935333 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36260f22-1889-4d44-a918-f8a0ccbe1326 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:46.937232 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-36260f22-1889-4d44-a918-f8a0ccbe1326 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:47.028813 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:47.083368 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36260f22-1889-4d44-a918-f8a0ccbe1326 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:16:47.088221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36260f22-1889-4d44-a918-f8a0ccbe1326 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:16:47.111322 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 660/1325] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:16:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:16:49.117237 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf2727a-a318-46cb-967b-efdcad935e1c None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:49.117790 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf2727a-a318-46cb-967b-efdcad935e1c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:49.117790 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf2727a-a318-46cb-967b-efdcad935e1c None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:49.119031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 666/1326] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:16:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:16:49.125361 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05ed2f-f394-4c2d-8c79-c04b20e95bae None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:49.125878 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05ed2f-f394-4c2d-8c79-c04b20e95bae None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:49.125977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05ed2f-f394-4c2d-8c79-c04b20e95bae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:49.196389 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-3f05ed2f-f394-4c2d-8c79-c04b20e95bae None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:16:49.197607 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3f05ed2f-f394-4c2d-8c79-c04b20e95bae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:49.301393 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 661/1327] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:16:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:16:49.585015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf2727a-a318-46cb-967b-efdcad935e1c None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:16:49.586709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-edf2727a-a318-46cb-967b-efdcad935e1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:49.650212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:49.700424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf2727a-a318-46cb-967b-efdcad935e1c None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:16:49.784367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592801ca-ea03-4eed-b460-155550cea078 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:16:49.785103 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592801ca-ea03-4eed-b460-155550cea078 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:49.785440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592801ca-ea03-4eed-b460-155550cea078 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:49.785780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592801ca-ea03-4eed-b460-155550cea078 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:49.787512 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-592801ca-ea03-4eed-b460-155550cea078 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:49.899851 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:49.902953 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592801ca-ea03-4eed-b460-155550cea078 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:16:49.904437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592801ca-ea03-4eed-b460-155550cea078 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:16:49.925833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 667/1328] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:16:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:16:51.927207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487bffc4-8f2d-4eae-ac53-5bc9f66937d0 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:51.927659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487bffc4-8f2d-4eae-ac53-5bc9f66937d0 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:51.927927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487bffc4-8f2d-4eae-ac53-5bc9f66937d0 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:51.929544 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 662/1329] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:16:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:16:51.936368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbe921c-2bd3-4f91-9496-cca99a443c84 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:51.936772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbe921c-2bd3-4f91-9496-cca99a443c84 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:51.936772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbe921c-2bd3-4f91-9496-cca99a443c84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:52.014491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-efbe921c-2bd3-4f91-9496-cca99a443c84 None None] MFA Rules not processed for user `4e36c8bcb5234cd29a8d01bb6f6d6d38`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:16:52.015722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-efbe921c-2bd3-4f91-9496-cca99a443c84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:52.138296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 668/1330] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:16:51 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:16:52.424072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487bffc4-8f2d-4eae-ac53-5bc9f66937d0 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:16:52.425784 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-487bffc4-8f2d-4eae-ac53-5bc9f66937d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:52.466670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:52.565142 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487bffc4-8f2d-4eae-ac53-5bc9f66937d0 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:16:52.643855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec59dde8-c8c8-42d1-80e8-4a23bc04da0e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:16:52.644489 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec59dde8-c8c8-42d1-80e8-4a23bc04da0e service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:52.644762 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec59dde8-c8c8-42d1-80e8-4a23bc04da0e service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:52.645031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec59dde8-c8c8-42d1-80e8-4a23bc04da0e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:52.646719 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ec59dde8-c8c8-42d1-80e8-4a23bc04da0e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:52.726284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:52.758865 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec59dde8-c8c8-42d1-80e8-4a23bc04da0e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:16:52.760318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec59dde8-c8c8-42d1-80e8-4a23bc04da0e service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:16:52.781007 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 663/1331] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:16:52 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:16:55.203761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12680a5-d77e-4c27-8046-6eedc50bba9c None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:55.204320 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12680a5-d77e-4c27-8046-6eedc50bba9c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:55.204320 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12680a5-d77e-4c27-8046-6eedc50bba9c None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:55.205687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 669/1332] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:16:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:16:55.211934 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be19baaa-e6dc-47dc-ba76-79aa1a2e5946 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:55.217749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be19baaa-e6dc-47dc-ba76-79aa1a2e5946 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:55.217749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be19baaa-e6dc-47dc-ba76-79aa1a2e5946 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:55.284658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-be19baaa-e6dc-47dc-ba76-79aa1a2e5946 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:16:55.286559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-be19baaa-e6dc-47dc-ba76-79aa1a2e5946 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:55.388931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 664/1333] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:16:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:16:55.665067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12680a5-d77e-4c27-8046-6eedc50bba9c None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:16:55.667417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b12680a5-d77e-4c27-8046-6eedc50bba9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:55.737938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:55.774308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12680a5-d77e-4c27-8046-6eedc50bba9c None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:16:55.854874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee35ed5c-1baa-49d1-95ac-0a22e10598fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:16:55.855472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee35ed5c-1baa-49d1-95ac-0a22e10598fb service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:55.855741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee35ed5c-1baa-49d1-95ac-0a22e10598fb service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:55.856002 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee35ed5c-1baa-49d1-95ac-0a22e10598fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:55.857673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ee35ed5c-1baa-49d1-95ac-0a22e10598fb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:55.973792 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee35ed5c-1baa-49d1-95ac-0a22e10598fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:16:55.975247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee35ed5c-1baa-49d1-95ac-0a22e10598fb service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:16:55.986510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:55.995373 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 670/1334] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:16:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:16:57.964361 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750a9c4e-27f7-4008-9fbf-63a28fd07b07 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:57.964890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750a9c4e-27f7-4008-9fbf-63a28fd07b07 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:57.964890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750a9c4e-27f7-4008-9fbf-63a28fd07b07 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:57.966230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 665/1335] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:16:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:16:57.971777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9e10ce-f5b2-445b-be12-4c58f0ecb0cf None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:57.972234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9e10ce-f5b2-445b-be12-4c58f0ecb0cf None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:57.972381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9e10ce-f5b2-445b-be12-4c58f0ecb0cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:58.047283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-8b9e10ce-f5b2-445b-be12-4c58f0ecb0cf None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:16:58.048548 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8b9e10ce-f5b2-445b-be12-4c58f0ecb0cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:58.163173 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 671/1336] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:16:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:16:58.456075 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750a9c4e-27f7-4008-9fbf-63a28fd07b07 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:16:58.457866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-750a9c4e-27f7-4008-9fbf-63a28fd07b07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:58.498798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:58.562653 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750a9c4e-27f7-4008-9fbf-63a28fd07b07 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:16:58.639301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063790b4-c4f7-4b58-8043-a314003ed368 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:16:58.639820 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063790b4-c4f7-4b58-8043-a314003ed368 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:16:58.640091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063790b4-c4f7-4b58-8043-a314003ed368 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:16:58.640382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063790b4-c4f7-4b58-8043-a314003ed368 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:16:58.641978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-063790b4-c4f7-4b58-8043-a314003ed368 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:16:58.723602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:16:58.748776 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063790b4-c4f7-4b58-8043-a314003ed368 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:16:58.750206 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063790b4-c4f7-4b58-8043-a314003ed368 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:16:58.768567 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 666/1337] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:16:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:17:01.338775 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0935337b-ce8a-43ab-855a-3a5841f1cf71 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:01.339341 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0935337b-ce8a-43ab-855a-3a5841f1cf71 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:01.339799 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0935337b-ce8a-43ab-855a-3a5841f1cf71 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:01.342103 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 672/1338] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:17:01 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:01.348701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc5d057-2402-425b-b45f-ca390df95ec1 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:01.349258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc5d057-2402-425b-b45f-ca390df95ec1 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:01.349398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc5d057-2402-425b-b45f-ca390df95ec1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:01.431080 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-3cc5d057-2402-425b-b45f-ca390df95ec1 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:01.432360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3cc5d057-2402-425b-b45f-ca390df95ec1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:01.545317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 667/1339] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:17:01 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:01.824588 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0935337b-ce8a-43ab-855a-3a5841f1cf71 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:17:01.826157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-0935337b-ce8a-43ab-855a-3a5841f1cf71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:01.901083 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:01.927703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0935337b-ce8a-43ab-855a-3a5841f1cf71 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:17:02.005004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c063bf7-7faf-45f3-b2a6-cf43eded6f55 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:17:02.005615 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c063bf7-7faf-45f3-b2a6-cf43eded6f55 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:02.005883 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c063bf7-7faf-45f3-b2a6-cf43eded6f55 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:02.006150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c063bf7-7faf-45f3-b2a6-cf43eded6f55 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:02.007775 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7c063bf7-7faf-45f3-b2a6-cf43eded6f55 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:02.126641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c063bf7-7faf-45f3-b2a6-cf43eded6f55 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:17:02.128188 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c063bf7-7faf-45f3-b2a6-cf43eded6f55 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:17:02.147595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 673/1340] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:17:01 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:17:06.406722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5785acf-9f79-49d2-aeee-53e2a226d97e None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:06.407253 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5785acf-9f79-49d2-aeee-53e2a226d97e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:06.407253 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5785acf-9f79-49d2-aeee-53e2a226d97e None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:06.411642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 668/1341] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:17:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:06.414976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cace067b-c60e-4708-9244-92f6958d70fa None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:06.415443 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cace067b-c60e-4708-9244-92f6958d70fa None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:06.415573 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cace067b-c60e-4708-9244-92f6958d70fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:06.421782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:06.488603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-cace067b-c60e-4708-9244-92f6958d70fa None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:06.489822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-cace067b-c60e-4708-9244-92f6958d70fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:06.594393 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 674/1342] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:17:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:06.881139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5785acf-9f79-49d2-aeee-53e2a226d97e None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:17:06.882721 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e5785acf-9f79-49d2-aeee-53e2a226d97e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:06.932713 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:06.990448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5785acf-9f79-49d2-aeee-53e2a226d97e None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:17:07.063454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b185fcbb-eef0-4788-85a8-c070f1b4a300 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:17:07.064053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b185fcbb-eef0-4788-85a8-c070f1b4a300 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:07.064349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b185fcbb-eef0-4788-85a8-c070f1b4a300 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:07.064614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b185fcbb-eef0-4788-85a8-c070f1b4a300 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:07.066197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-b185fcbb-eef0-4788-85a8-c070f1b4a300 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:07.164009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:07.174974 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b185fcbb-eef0-4788-85a8-c070f1b4a300 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:17:07.176429 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b185fcbb-eef0-4788-85a8-c070f1b4a300 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:17:07.194325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 669/1343] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:17:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:17:10.052025 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f932093-780c-4afe-889d-bf9eaf9c5319 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:10.052617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f932093-780c-4afe-889d-bf9eaf9c5319 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:10.052729 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f932093-780c-4afe-889d-bf9eaf9c5319 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:10.054019 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 675/1344] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:17:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:10.060141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74968655-cf4a-46cc-8d30-cf491b0530a1 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:10.060609 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74968655-cf4a-46cc-8d30-cf491b0530a1 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:10.060701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74968655-cf4a-46cc-8d30-cf491b0530a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:10.131192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-74968655-cf4a-46cc-8d30-cf491b0530a1 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:10.132423 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-74968655-cf4a-46cc-8d30-cf491b0530a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:10.242644 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 670/1345] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:17:10 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:10.523146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f932093-780c-4afe-889d-bf9eaf9c5319 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:17:10.524730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9f932093-780c-4afe-889d-bf9eaf9c5319 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:10.625246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:10.642049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f932093-780c-4afe-889d-bf9eaf9c5319 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:17:10.741101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693fa20c-01eb-4806-9547-b30df79d6b7a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:17:10.741733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693fa20c-01eb-4806-9547-b30df79d6b7a service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:10.742004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693fa20c-01eb-4806-9547-b30df79d6b7a service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:10.742270 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693fa20c-01eb-4806-9547-b30df79d6b7a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:10.743904 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-693fa20c-01eb-4806-9547-b30df79d6b7a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:10.862895 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693fa20c-01eb-4806-9547-b30df79d6b7a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:17:10.864437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693fa20c-01eb-4806-9547-b30df79d6b7a service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:17:10.883297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 676/1346] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:17:10 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:17:15.077613 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79f7f87-e409-4367-bb9f-7627a5d0a35f None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:15.078160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79f7f87-e409-4367-bb9f-7627a5d0a35f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:15.078160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79f7f87-e409-4367-bb9f-7627a5d0a35f None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:15.079578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 671/1347] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:17:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:15.086313 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbec8cab-46c6-4227-b936-2f7a0ea92c8a None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:15.087341 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbec8cab-46c6-4227-b936-2f7a0ea92c8a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:15.087443 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbec8cab-46c6-4227-b936-2f7a0ea92c8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:15.106124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:15.171603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-cbec8cab-46c6-4227-b936-2f7a0ea92c8a None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:15.172852 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-cbec8cab-46c6-4227-b936-2f7a0ea92c8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:15.291046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 677/1348] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:17:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:15.576602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79f7f87-e409-4367-bb9f-7627a5d0a35f None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:17:15.578150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-f79f7f87-e409-4367-bb9f-7627a5d0a35f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:15.637691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:15.685597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79f7f87-e409-4367-bb9f-7627a5d0a35f None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:17:15.772938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdde991-9374-4cbb-bbf5-ab182e6309f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:17:15.774084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdde991-9374-4cbb-bbf5-ab182e6309f3 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:15.776022 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdde991-9374-4cbb-bbf5-ab182e6309f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:15.776022 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdde991-9374-4cbb-bbf5-ab182e6309f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:15.777830 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3cdde991-9374-4cbb-bbf5-ab182e6309f3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:15.900371 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:15.904092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdde991-9374-4cbb-bbf5-ab182e6309f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:17:15.905598 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdde991-9374-4cbb-bbf5-ab182e6309f3 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:17:15.923232 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 672/1349] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:17:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:17:21.078972 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72276a88-a729-49dd-8be4-0eec7f58419f None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:21.079789 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72276a88-a729-49dd-8be4-0eec7f58419f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:21.080258 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72276a88-a729-49dd-8be4-0eec7f58419f None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:21.081748 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 678/1350] 10.222.0.47 () {54 vars in 921 bytes} [Sun Aug 21 01:17:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:21.091482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d1ad99-6573-4f6b-b49d-2d1ceb6e64e5 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:21.091996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d1ad99-6573-4f6b-b49d-2d1ceb6e64e5 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:21.092261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d1ad99-6573-4f6b-b49d-2d1ceb6e64e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:21.213150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-41d1ad99-6573-4f6b-b49d-2d1ceb6e64e5 None None] MFA Rules not processed for user `972b0f1dbfef4d85a6dfc4c5fd839bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:21.215719 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-41d1ad99-6573-4f6b-b49d-2d1ceb6e64e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:21.354645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 673/1351] 10.222.0.47 () {58 vars in 1018 bytes} [Sun Aug 21 01:17:21 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:25.287704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=59 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:26.107771 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe8d4fa-5622-4ecb-bcc2-2dc95c7a0ca2 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:26.108263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe8d4fa-5622-4ecb-bcc2-2dc95c7a0ca2 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:26.108642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe8d4fa-5622-4ecb-bcc2-2dc95c7a0ca2 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:26.110723 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 679/1352] 10.222.0.47 () {54 vars in 921 bytes} [Sun Aug 21 01:17:26 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:26.118259 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f3d607-5726-48b6-9927-d42db690a777 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:26.118259 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f3d607-5726-48b6-9927-d42db690a777 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:26.119696 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f3d607-5726-48b6-9927-d42db690a777 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:26.198303 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:26.232697 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-84f3d607-5726-48b6-9927-d42db690a777 None None] MFA Rules not processed for user `972b0f1dbfef4d85a6dfc4c5fd839bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:26.234794 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-84f3d607-5726-48b6-9927-d42db690a777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:26.411691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 674/1353] 10.222.0.47 () {58 vars in 1018 bytes} [Sun Aug 21 01:17:26 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 295 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:27.908261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d95dd0-638d-4687-a48a-85a11643188f None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:27.908761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d95dd0-638d-4687-a48a-85a11643188f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:27.908761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d95dd0-638d-4687-a48a-85a11643188f None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:27.910063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 680/1354] 10.222.0.47 () {54 vars in 921 bytes} [Sun Aug 21 01:17:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:27.915362 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c346c4f-49b5-428b-b465-de7a35791fb4 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:27.915798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c346c4f-49b5-428b-b465-de7a35791fb4 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:27.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c346c4f-49b5-428b-b465-de7a35791fb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:28.001332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-3c346c4f-49b5-428b-b465-de7a35791fb4 None None] MFA Rules not processed for user `972b0f1dbfef4d85a6dfc4c5fd839bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:28.002577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3c346c4f-49b5-428b-b465-de7a35791fb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:28.053409 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:28.124386 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 675/1355] 10.222.0.47 () {58 vars in 1018 bytes} [Sun Aug 21 01:17:27 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:31.649440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90eedf3-ab19-4e87-9f58-c850e40014aa None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:31.649440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90eedf3-ab19-4e87-9f58-c850e40014aa None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:31.649440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90eedf3-ab19-4e87-9f58-c850e40014aa None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:31.651157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 681/1356] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:17:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:31.655961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd37821-8ef7-416d-9e11-4bb8654e0818 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:31.656458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd37821-8ef7-416d-9e11-4bb8654e0818 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:31.656560 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd37821-8ef7-416d-9e11-4bb8654e0818 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:31.730640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-3fd37821-8ef7-416d-9e11-4bb8654e0818 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:31.732453 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3fd37821-8ef7-416d-9e11-4bb8654e0818 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:31.848475 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 676/1357] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:17:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:17:32.160940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2d60d8-bfc4-46db-be52-2246e48d2473 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:32.161471 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2d60d8-bfc4-46db-be52-2246e48d2473 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:32.161910 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2d60d8-bfc4-46db-be52-2246e48d2473 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:32.163987 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 682/1358] 10.222.0.47 () {54 vars in 962 bytes} [Sun Aug 21 01:17:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:32.172392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4bc7c6-4fa9-420b-877a-ea3e4ae5fceb None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:32.173071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4bc7c6-4fa9-420b-877a-ea3e4ae5fceb None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:32.173887 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4bc7c6-4fa9-420b-877a-ea3e4ae5fceb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:32.196242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:32.278117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-3c4bc7c6-4fa9-420b-877a-ea3e4ae5fceb None None] MFA Rules not processed for user `fabdcdc25d1d4ef6bc66bd6cadecaaf8`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:32.280222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3c4bc7c6-4fa9-420b-877a-ea3e4ae5fceb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:32.420389 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 677/1359] 10.222.0.47 () {58 vars in 1059 bytes} [Sun Aug 21 01:17:32 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:17:32.428025 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2d60d8-bfc4-46db-be52-2246e48d2473 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:17:32.430089 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8a2d60d8-bfc4-46db-be52-2246e48d2473 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:32.588648 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2d60d8-bfc4-46db-be52-2246e48d2473 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:17:32.665241 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:32.708205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6c8a28-17bd-4825-bf4b-edc500c9e5e4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:17:32.708815 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6c8a28-17bd-4825-bf4b-edc500c9e5e4 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:32.709093 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6c8a28-17bd-4825-bf4b-edc500c9e5e4 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:32.709356 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6c8a28-17bd-4825-bf4b-edc500c9e5e4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:32.710984 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5e6c8a28-17bd-4825-bf4b-edc500c9e5e4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:32.803051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6c8a28-17bd-4825-bf4b-edc500c9e5e4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:17:32.805229 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6c8a28-17bd-4825-bf4b-edc500c9e5e4 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:17:32.823999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 683/1360] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:17:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:17:34.955378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0730363-2b6a-40f8-81bf-25990c50b9d4 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:34.955911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0730363-2b6a-40f8-81bf-25990c50b9d4 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:34.955911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0730363-2b6a-40f8-81bf-25990c50b9d4 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:34.957363 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 678/1361] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:17:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:34.962848 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e141598-9bbd-4cc9-b922-6899b954ec05 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:34.963277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e141598-9bbd-4cc9-b922-6899b954ec05 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:34.963369 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e141598-9bbd-4cc9-b922-6899b954ec05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:35.030792 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-5e141598-9bbd-4cc9-b922-6899b954ec05 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:35.032010 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5e141598-9bbd-4cc9-b922-6899b954ec05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:35.073347 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:35.140094 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 684/1362] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:17:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:35.433136 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0730363-2b6a-40f8-81bf-25990c50b9d4 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:17:35.434666 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e0730363-2b6a-40f8-81bf-25990c50b9d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:35.508902 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:35.627997 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0730363-2b6a-40f8-81bf-25990c50b9d4 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:17:35.764285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece0f22b-9c80-4001-b65e-e917e7e4b79b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:17:35.765000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece0f22b-9c80-4001-b65e-e917e7e4b79b service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:35.765306 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece0f22b-9c80-4001-b65e-e917e7e4b79b service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:35.765864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece0f22b-9c80-4001-b65e-e917e7e4b79b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:35.767614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ece0f22b-9c80-4001-b65e-e917e7e4b79b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:35.793319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:35.873695 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece0f22b-9c80-4001-b65e-e917e7e4b79b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:17:35.875488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece0f22b-9c80-4001-b65e-e917e7e4b79b service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:17:35.910903 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 679/1363] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:17:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 478 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:17:38.302141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2b434e-280f-413c-a054-f2c185830567 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:38.302746 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2b434e-280f-413c-a054-f2c185830567 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:38.302746 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2b434e-280f-413c-a054-f2c185830567 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:38.304671 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 685/1364] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:17:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:38.310828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be0de94-a121-4abd-a929-64f5c5f10899 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:38.310828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be0de94-a121-4abd-a929-64f5c5f10899 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:38.311703 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be0de94-a121-4abd-a929-64f5c5f10899 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:38.395718 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-0be0de94-a121-4abd-a929-64f5c5f10899 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:38.397063 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-0be0de94-a121-4abd-a929-64f5c5f10899 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:38.519373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:38.523373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 680/1365] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:17:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:38.814792 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c764d20e-36c1-4c55-8397-b5f7a26004f2 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:38.815331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c764d20e-36c1-4c55-8397-b5f7a26004f2 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:38.815759 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c764d20e-36c1-4c55-8397-b5f7a26004f2 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:38.817557 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 686/1366] 10.222.0.47 () {54 vars in 962 bytes} [Sun Aug 21 01:17:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:38.823853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5c72a2-14c8-4f05-b926-69176acf5367 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:38.824397 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5c72a2-14c8-4f05-b926-69176acf5367 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:38.824857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5c72a2-14c8-4f05-b926-69176acf5367 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:38.913289 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-be5c72a2-14c8-4f05-b926-69176acf5367 None None] MFA Rules not processed for user `fabdcdc25d1d4ef6bc66bd6cadecaaf8`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:38.914835 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-be5c72a2-14c8-4f05-b926-69176acf5367 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:39.049576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 681/1367] 10.222.0.47 () {58 vars in 1059 bytes} [Sun Aug 21 01:17:38 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:39.057508 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c764d20e-36c1-4c55-8397-b5f7a26004f2 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:17:39.059105 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-c764d20e-36c1-4c55-8397-b5f7a26004f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:39.220753 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c764d20e-36c1-4c55-8397-b5f7a26004f2 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:17:39.244033 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:39.346510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00f05b9-7c03-488a-9ce8-194865d10e69 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:17:39.347290 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00f05b9-7c03-488a-9ce8-194865d10e69 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:39.347659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00f05b9-7c03-488a-9ce8-194865d10e69 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:39.347791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00f05b9-7c03-488a-9ce8-194865d10e69 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:39.349472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f00f05b9-7c03-488a-9ce8-194865d10e69 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:39.446199 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00f05b9-7c03-488a-9ce8-194865d10e69 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:17:39.447787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00f05b9-7c03-488a-9ce8-194865d10e69 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:17:39.467137 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 687/1368] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:17:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:17:40.218571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acff3a8f-ec33-44ae-b907-ea630ca84572 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:40.219354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acff3a8f-ec33-44ae-b907-ea630ca84572 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:40.219354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acff3a8f-ec33-44ae-b907-ea630ca84572 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:40.224696 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 682/1369] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:17:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:40.228263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1956a790-9f50-470d-9809-3d7687b497e3 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:40.228556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1956a790-9f50-470d-9809-3d7687b497e3 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:40.228832 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1956a790-9f50-470d-9809-3d7687b497e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:40.278466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:40.317599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-1956a790-9f50-470d-9809-3d7687b497e3 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:40.318815 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-1956a790-9f50-470d-9809-3d7687b497e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:40.435809 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 688/1370] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:17:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:41.036096 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb410dbe-32ea-43c0-a993-d46b019102bc None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:41.036096 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb410dbe-32ea-43c0-a993-d46b019102bc None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:41.036096 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb410dbe-32ea-43c0-a993-d46b019102bc None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:41.039761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 683/1371] 10.222.0.47 () {54 vars in 964 bytes} [Sun Aug 21 01:17:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:41.044028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ab468e-a304-4cad-8098-ac23f39beffc None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:41.044028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ab468e-a304-4cad-8098-ac23f39beffc None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:41.044028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ab468e-a304-4cad-8098-ac23f39beffc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:41.137421 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-30ab468e-a304-4cad-8098-ac23f39beffc None None] MFA Rules not processed for user `b953036b8b774f25bda59cec96b21c27`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:41.138777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-30ab468e-a304-4cad-8098-ac23f39beffc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:41.199926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:41.234463 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 689/1372] 10.222.0.47 () {58 vars in 1061 bytes} [Sun Aug 21 01:17:41 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:41.242374 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb410dbe-32ea-43c0-a993-d46b019102bc None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:17:41.244489 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-cb410dbe-32ea-43c0-a993-d46b019102bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:41.305137 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:41.379194 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb410dbe-32ea-43c0-a993-d46b019102bc None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:17:41.471855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e7a629-42af-47c3-b01d-089300eeb8d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:17:41.472501 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e7a629-42af-47c3-b01d-089300eeb8d5 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:41.472782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e7a629-42af-47c3-b01d-089300eeb8d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:41.473051 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e7a629-42af-47c3-b01d-089300eeb8d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:41.474744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-71e7a629-42af-47c3-b01d-089300eeb8d5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:41.591136 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:41.606882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e7a629-42af-47c3-b01d-089300eeb8d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:17:41.607898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e7a629-42af-47c3-b01d-089300eeb8d5 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:17:41.628366 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 684/1373] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:17:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:17:41.640494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f98067a-86bd-4756-8a67-1d39cbefb5fe None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:41.640960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f98067a-86bd-4756-8a67-1d39cbefb5fe None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:41.641210 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f98067a-86bd-4756-8a67-1d39cbefb5fe None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:41.642573 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 690/1374] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:17:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:17:41.648545 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43e4310-1a7b-445b-a3be-02f41752ca82 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:41.648830 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43e4310-1a7b-445b-a3be-02f41752ca82 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:41.649218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43e4310-1a7b-445b-a3be-02f41752ca82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:41.724325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-c43e4310-1a7b-445b-a3be-02f41752ca82 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:41.726374 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c43e4310-1a7b-445b-a3be-02f41752ca82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:41.861266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 685/1375] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:17:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:42.155247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f98067a-86bd-4756-8a67-1d39cbefb5fe None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:17:42.156857 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-3f98067a-86bd-4756-8a67-1d39cbefb5fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:42.311914 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f98067a-86bd-4756-8a67-1d39cbefb5fe None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:17:42.365254 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:42.428833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21e0320-006f-4005-961a-1d863eb49561 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:17:42.429446 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21e0320-006f-4005-961a-1d863eb49561 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:42.429721 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21e0320-006f-4005-961a-1d863eb49561 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:42.430030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21e0320-006f-4005-961a-1d863eb49561 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:42.432486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a21e0320-006f-4005-961a-1d863eb49561 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:42.528821 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21e0320-006f-4005-961a-1d863eb49561 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:17:42.530542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21e0320-006f-4005-961a-1d863eb49561 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:17:42.550046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 691/1376] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:17:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:17:44.641274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9228ea9e-1798-4eae-a5e8-6d698227fc26 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:44.641811 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9228ea9e-1798-4eae-a5e8-6d698227fc26 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:44.641811 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9228ea9e-1798-4eae-a5e8-6d698227fc26 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:44.643067 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 686/1377] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:17:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:44.648824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b4c8fa-ac62-41b8-b647-13f83a0323cc None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:44.649226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b4c8fa-ac62-41b8-b647-13f83a0323cc None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:44.649375 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b4c8fa-ac62-41b8-b647-13f83a0323cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:44.716397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f7b4c8fa-ac62-41b8-b647-13f83a0323cc None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:44.717678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f7b4c8fa-ac62-41b8-b647-13f83a0323cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:44.724632 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:44.842021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 692/1378] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:17:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:45.141317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9228ea9e-1798-4eae-a5e8-6d698227fc26 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:17:45.142908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9228ea9e-1798-4eae-a5e8-6d698227fc26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:45.238839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:45.351415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9228ea9e-1798-4eae-a5e8-6d698227fc26 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:17:45.537434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3795d53-471d-4c4d-8781-498d273da252 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:17:45.538517 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3795d53-471d-4c4d-8781-498d273da252 service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:45.538794 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3795d53-471d-4c4d-8781-498d273da252 service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:45.539063 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3795d53-471d-4c4d-8781-498d273da252 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:45.541165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-d3795d53-471d-4c4d-8781-498d273da252 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:45.569946 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:45.680602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3795d53-471d-4c4d-8781-498d273da252 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:17:45.683589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3795d53-471d-4c4d-8781-498d273da252 service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:17:45.711430 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 687/1379] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:17:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 570 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:17:47.952216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e91a8c-0132-4a68-942a-4f971ad410e5 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:47.952770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e91a8c-0132-4a68-942a-4f971ad410e5 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:47.952770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e91a8c-0132-4a68-942a-4f971ad410e5 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:47.954285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 693/1380] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:17:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:47.959598 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22765a1e-077a-4a4d-bc83-e44f9618d1cc None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:47.960033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22765a1e-077a-4a4d-bc83-e44f9618d1cc None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:47.960245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22765a1e-077a-4a4d-bc83-e44f9618d1cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:48.033616 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-22765a1e-077a-4a4d-bc83-e44f9618d1cc None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:48.034795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-22765a1e-077a-4a4d-bc83-e44f9618d1cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:48.140389 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:48.148177 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 688/1381] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:17:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:48.456139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e91a8c-0132-4a68-942a-4f971ad410e5 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:17:48.457959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-77e91a8c-0132-4a68-942a-4f971ad410e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:48.623426 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:48.635650 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e91a8c-0132-4a68-942a-4f971ad410e5 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:17:48.775381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cf51e4-0891-428b-8539-22c308cfe544 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:17:48.776001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cf51e4-0891-428b-8539-22c308cfe544 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:48.776305 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cf51e4-0891-428b-8539-22c308cfe544 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:48.776574 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cf51e4-0891-428b-8539-22c308cfe544 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:48.778555 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-89cf51e4-0891-428b-8539-22c308cfe544 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:48.883670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cf51e4-0891-428b-8539-22c308cfe544 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:17:48.885255 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cf51e4-0891-428b-8539-22c308cfe544 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:17:48.899581 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:48.910253 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 694/1382] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:17:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 454 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:17:51.161669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df729d4a-f432-4d18-892b-2e29c1f3fbda None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:51.165855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df729d4a-f432-4d18-892b-2e29c1f3fbda None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:51.165855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df729d4a-f432-4d18-892b-2e29c1f3fbda None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:51.165855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 689/1383] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:17:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:51.170130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3693c87-bc41-4ca4-9a92-a83f350c6fdf None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:51.170605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3693c87-bc41-4ca4-9a92-a83f350c6fdf None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:51.170894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3693c87-bc41-4ca4-9a92-a83f350c6fdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:51.254843 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-a3693c87-bc41-4ca4-9a92-a83f350c6fdf None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:51.256728 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a3693c87-bc41-4ca4-9a92-a83f350c6fdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:51.406171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 695/1384] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:17:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:51.691254 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df729d4a-f432-4d18-892b-2e29c1f3fbda None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:17:51.693649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-df729d4a-f432-4d18-892b-2e29c1f3fbda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:51.893857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df729d4a-f432-4d18-892b-2e29c1f3fbda None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:17:51.982990 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:52.044711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f7d406-3ffd-44ca-93e2-dc7d5eabba31 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:17:52.045367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f7d406-3ffd-44ca-93e2-dc7d5eabba31 service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:52.045652 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f7d406-3ffd-44ca-93e2-dc7d5eabba31 service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:52.045920 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f7d406-3ffd-44ca-93e2-dc7d5eabba31 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:52.047596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-06f7d406-3ffd-44ca-93e2-dc7d5eabba31 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:52.156298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f7d406-3ffd-44ca-93e2-dc7d5eabba31 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:17:52.158010 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f7d406-3ffd-44ca-93e2-dc7d5eabba31 service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:17:52.180935 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 690/1385] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:17:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 491 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:17:54.259648 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09722c3f-0773-4f53-82e0-e3967bceae99 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:54.260251 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09722c3f-0773-4f53-82e0-e3967bceae99 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:54.260251 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09722c3f-0773-4f53-82e0-e3967bceae99 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:54.261626 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 696/1386] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:17:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:54.267137 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284df222-6098-4b0a-a511-e0a98943078e None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:54.267552 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284df222-6098-4b0a-a511-e0a98943078e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:54.267642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284df222-6098-4b0a-a511-e0a98943078e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:54.343880 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-284df222-6098-4b0a-a511-e0a98943078e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:54.345424 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-284df222-6098-4b0a-a511-e0a98943078e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:54.382718 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:54.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 691/1387] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:17:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:54.797321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09722c3f-0773-4f53-82e0-e3967bceae99 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:17:54.799053 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-09722c3f-0773-4f53-82e0-e3967bceae99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:54.883132 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:54.985328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09722c3f-0773-4f53-82e0-e3967bceae99 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:17:55.137792 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b43658-1b78-4a61-a3f3-ba0804e52f4a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:17:55.138480 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b43658-1b78-4a61-a3f3-ba0804e52f4a service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:55.138755 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b43658-1b78-4a61-a3f3-ba0804e52f4a service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:55.139039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b43658-1b78-4a61-a3f3-ba0804e52f4a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:55.140860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-45b43658-1b78-4a61-a3f3-ba0804e52f4a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:55.203751 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:55.281479 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b43658-1b78-4a61-a3f3-ba0804e52f4a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:17:55.283924 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b43658-1b78-4a61-a3f3-ba0804e52f4a service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:17:55.316796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 697/1388] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:17:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 520 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:17:57.167995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa8e988-5302-4d35-97a2-7ff83ee8e3a2 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:57.168614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa8e988-5302-4d35-97a2-7ff83ee8e3a2 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:57.168614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa8e988-5302-4d35-97a2-7ff83ee8e3a2 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:57.170016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 692/1389] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:17:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:57.180060 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee1187a-f0b4-42a6-9095-82a0611ca132 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:57.180629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee1187a-f0b4-42a6-9095-82a0611ca132 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:57.181049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee1187a-f0b4-42a6-9095-82a0611ca132 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:57.278274 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-cee1187a-f0b4-42a6-9095-82a0611ca132 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:57.281393 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-cee1187a-f0b4-42a6-9095-82a0611ca132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:57.390518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f640420a-0a80-41c1-ad6b-877ed351de07 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:57.390901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f640420a-0a80-41c1-ad6b-877ed351de07 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:57.391245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f640420a-0a80-41c1-ad6b-877ed351de07 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:57.392891 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 693/1390] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:17:57 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:57.398443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40521d85-36f3-4cfa-b156-58320279e77f None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:57.398719 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40521d85-36f3-4cfa-b156-58320279e77f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:57.398994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40521d85-36f3-4cfa-b156-58320279e77f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:57.400926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 698/1391] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:17:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:57.477355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-40521d85-36f3-4cfa-b156-58320279e77f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:57.478562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-40521d85-36f3-4cfa-b156-58320279e77f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:57.627837 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 694/1392] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:17:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:17:57.722201 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bf4410-786d-4874-bcd2-dda464bd699e None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:57.722513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bf4410-786d-4874-bcd2-dda464bd699e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:57.722745 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bf4410-786d-4874-bcd2-dda464bd699e None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:57.725460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 699/1393] 10.222.0.47 () {54 vars in 967 bytes} [Sun Aug 21 01:17:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:57.730765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad43b5f-fc31-4dd7-81b7-699bdece96ae None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:57.736731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad43b5f-fc31-4dd7-81b7-699bdece96ae None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:57.736731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad43b5f-fc31-4dd7-81b7-699bdece96ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:57.750099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:57.837807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-2ad43b5f-fc31-4dd7-81b7-699bdece96ae None None] MFA Rules not processed for user `a6365b2328f34ab9b1b73ececbf5cc30`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:57.839042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-2ad43b5f-fc31-4dd7-81b7-699bdece96ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:57.925071 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bf4410-786d-4874-bcd2-dda464bd699e None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:17:57.927780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-51bf4410-786d-4874-bcd2-dda464bd699e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:57.948483 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 695/1394] 10.222.0.47 () {58 vars in 1064 bytes} [Sun Aug 21 01:17:57 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:57.948682 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:57.956708 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad43b5f-fc31-4dd7-81b7-699bdece96ae None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:17:57.966734 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad43b5f-fc31-4dd7-81b7-699bdece96ae None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:17:58.089370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274ceef8-ec6e-49a1-9048-118130ab9767 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6365b2328f34ab9b1b73ececbf5cc30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:17:58.090143 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274ceef8-ec6e-49a1-9048-118130ab9767 service glance] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:58.090505 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274ceef8-ec6e-49a1-9048-118130ab9767 service glance] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:58.090779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274ceef8-ec6e-49a1-9048-118130ab9767 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:58.093023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-274ceef8-ec6e-49a1-9048-118130ab9767 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:58.119406 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bf4410-786d-4874-bcd2-dda464bd699e None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:17:58.131972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:58.239240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274ceef8-ec6e-49a1-9048-118130ab9767 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:17:58.241400 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274ceef8-ec6e-49a1-9048-118130ab9767 service glance] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:17:58.244925 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:58.264829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 696/1395] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:17:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:17:58.275628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb89dd9-f57e-4ebd-be70-1c399927311c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:17:58.276574 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb89dd9-f57e-4ebd-be70-1c399927311c service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:58.276849 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb89dd9-f57e-4ebd-be70-1c399927311c service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:58.277118 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb89dd9-f57e-4ebd-be70-1c399927311c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:58.278743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8bb89dd9-f57e-4ebd-be70-1c399927311c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:58.388264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb89dd9-f57e-4ebd-be70-1c399927311c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:17:58.390115 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb89dd9-f57e-4ebd-be70-1c399927311c service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:17:58.409855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 700/1396] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:17:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 486 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:17:59.217446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385c4253-73ac-4e93-9837-95f05315731d None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:59.217746 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385c4253-73ac-4e93-9837-95f05315731d None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:59.218013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385c4253-73ac-4e93-9837-95f05315731d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:59.296446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-385c4253-73ac-4e93-9837-95f05315731d None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:59.298203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-385c4253-73ac-4e93-9837-95f05315731d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:59.391907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:59.398475 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 697/1397] 10.222.0.47 () {58 vars in 1009 bytes} [Sun Aug 21 01:17:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:59.465700 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6936bfd-4d44-42f5-aa6e-76cc2b13d572 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:59.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6936bfd-4d44-42f5-aa6e-76cc2b13d572 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:59.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6936bfd-4d44-42f5-aa6e-76cc2b13d572 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:59.467605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 701/1398] 10.222.0.47 () {54 vars in 943 bytes} [Sun Aug 21 01:17:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:17:59.474418 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af325b5-4789-4ff2-b3b3-7a912eb343b4 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:17:59.474680 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af325b5-4789-4ff2-b3b3-7a912eb343b4 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:17:59.474943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af325b5-4789-4ff2-b3b3-7a912eb343b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:17:59.565514 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-0af325b5-4789-4ff2-b3b3-7a912eb343b4 None None] MFA Rules not processed for user `06a5972a55824711b06670b925322739`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:17:59.566863 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-0af325b5-4789-4ff2-b3b3-7a912eb343b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:59.691982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 698/1399] 10.222.0.47 () {58 vars in 1040 bytes} [Sun Aug 21 01:17:59 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:17:59.702224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6936bfd-4d44-42f5-aa6e-76cc2b13d572 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:17:59.703819 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b6936bfd-4d44-42f5-aa6e-76cc2b13d572 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:17:59.835827 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:17:59.883670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6936bfd-4d44-42f5-aa6e-76cc2b13d572 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:00.030778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe25137-d8f4-4bb4-8c7b-0b9634e14f68 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:00.031320 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe25137-d8f4-4bb4-8c7b-0b9634e14f68 service swift] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:00.031582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe25137-d8f4-4bb4-8c7b-0b9634e14f68 service swift] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:00.031890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe25137-d8f4-4bb4-8c7b-0b9634e14f68 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:00.033568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9fe25137-d8f4-4bb4-8c7b-0b9634e14f68 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:00.144571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe25137-d8f4-4bb4-8c7b-0b9634e14f68 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:00.146691 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe25137-d8f4-4bb4-8c7b-0b9634e14f68 service swift] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:00.151008 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:00.158284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 702/1400] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:17:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 21 01:18:00.694526 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be696bf-40a2-430d-8d55-1afb7314c3c9 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:00.695208 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be696bf-40a2-430d-8d55-1afb7314c3c9 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:00.695208 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be696bf-40a2-430d-8d55-1afb7314c3c9 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:00.696559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 699/1401] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:18:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:18:00.701979 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b30ce01-03ba-4ea3-b316-b723f86559a3 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:00.702468 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b30ce01-03ba-4ea3-b316-b723f86559a3 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:00.702859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b30ce01-03ba-4ea3-b316-b723f86559a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:00.802475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-8b30ce01-03ba-4ea3-b316-b723f86559a3 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:00.803877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8b30ce01-03ba-4ea3-b316-b723f86559a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:00.932072 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 703/1402] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:18:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:01.250206 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be696bf-40a2-430d-8d55-1afb7314c3c9 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:01.251934 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-5be696bf-40a2-430d-8d55-1afb7314c3c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:01.329676 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:01.463142 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be696bf-40a2-430d-8d55-1afb7314c3c9 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:01.612833 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55b2ef1-9571-4256-b497-0135d2559970 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:01.613362 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55b2ef1-9571-4256-b497-0135d2559970 service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:01.613639 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55b2ef1-9571-4256-b497-0135d2559970 service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:01.613905 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55b2ef1-9571-4256-b497-0135d2559970 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:01.616010 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c55b2ef1-9571-4256-b497-0135d2559970 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:01.724388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55b2ef1-9571-4256-b497-0135d2559970 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:01.726864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55b2ef1-9571-4256-b497-0135d2559970 service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:01.745874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 700/1403] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 496 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:02.029179 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:04.154296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8212dfcf-272d-4a1a-920e-c1e8c9e4fff0 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:04.154879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8212dfcf-272d-4a1a-920e-c1e8c9e4fff0 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:04.155101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8212dfcf-272d-4a1a-920e-c1e8c9e4fff0 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:04.156506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 704/1404] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:18:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:04.163096 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a05c70-3101-4a7a-848f-130fe8b7e8f6 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:04.164218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a05c70-3101-4a7a-848f-130fe8b7e8f6 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:04.164218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a05c70-3101-4a7a-848f-130fe8b7e8f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:04.240073 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-61a05c70-3101-4a7a-848f-130fe8b7e8f6 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:04.241334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-61a05c70-3101-4a7a-848f-130fe8b7e8f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:04.353838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:04.362675 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 701/1405] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:18:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:04.665420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8212dfcf-272d-4a1a-920e-c1e8c9e4fff0 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:04.666940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8212dfcf-272d-4a1a-920e-c1e8c9e4fff0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:04.776515 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:04.873109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8212dfcf-272d-4a1a-920e-c1e8c9e4fff0 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:05.022487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0620c75-863c-403a-ba9b-d39139d62e08 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:05.023102 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0620c75-863c-403a-ba9b-d39139d62e08 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:05.023374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0620c75-863c-403a-ba9b-d39139d62e08 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:05.023638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0620c75-863c-403a-ba9b-d39139d62e08 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:05.025296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a0620c75-863c-403a-ba9b-d39139d62e08 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:05.075779 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:05.139365 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0620c75-863c-403a-ba9b-d39139d62e08 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:05.141123 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0620c75-863c-403a-ba9b-d39139d62e08 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:05.163439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 705/1406] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 498 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:07.371305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62314d4f-5289-4af8-8b32-f8ac8d6c315a None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:07.372027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62314d4f-5289-4af8-8b32-f8ac8d6c315a None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:07.372027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62314d4f-5289-4af8-8b32-f8ac8d6c315a None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:07.373258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 702/1407] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:18:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:07.379407 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a15c7ab-9717-49f4-a684-bd09f10d00f5 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:07.381142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a15c7ab-9717-49f4-a684-bd09f10d00f5 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:07.381630 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a15c7ab-9717-49f4-a684-bd09f10d00f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:07.454146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-4a15c7ab-9717-49f4-a684-bd09f10d00f5 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:07.455387 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-4a15c7ab-9717-49f4-a684-bd09f10d00f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:07.591287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 706/1408] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:18:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:07.902942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62314d4f-5289-4af8-8b32-f8ac8d6c315a None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:07.904628 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-62314d4f-5289-4af8-8b32-f8ac8d6c315a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:08.111677 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62314d4f-5289-4af8-8b32-f8ac8d6c315a None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:08.203238 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:08.256864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f3d28b-4d25-4fe9-9bbd-e77f8db814e8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:08.257480 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f3d28b-4d25-4fe9-9bbd-e77f8db814e8 service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:08.257761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f3d28b-4d25-4fe9-9bbd-e77f8db814e8 service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:08.258034 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f3d28b-4d25-4fe9-9bbd-e77f8db814e8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:08.259671 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-27f3d28b-4d25-4fe9-9bbd-e77f8db814e8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:08.383559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f3d28b-4d25-4fe9-9bbd-e77f8db814e8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:08.385292 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f3d28b-4d25-4fe9-9bbd-e77f8db814e8 service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:08.405197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 703/1409] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 503 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:17.589074 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=93 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:18.402902 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=65 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:24.109259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42751200-ac06-47d0-9952-0643793acf18 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:24.109843 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42751200-ac06-47d0-9952-0643793acf18 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:24.109843 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42751200-ac06-47d0-9952-0643793acf18 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:24.111158 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 707/1410] 10.222.0.47 () {54 vars in 918 bytes} [Sun Aug 21 01:18:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:24.116398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56cc2a0-2a2c-4d50-9029-0f1021bbc76e None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:24.116855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56cc2a0-2a2c-4d50-9029-0f1021bbc76e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:24.116966 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56cc2a0-2a2c-4d50-9029-0f1021bbc76e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:24.193361 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-e56cc2a0-2a2c-4d50-9029-0f1021bbc76e None None] MFA Rules not processed for user `972b0f1dbfef4d85a6dfc4c5fd839bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:24.194566 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e56cc2a0-2a2c-4d50-9029-0f1021bbc76e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:24.320376 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 704/1411] 10.222.0.47 () {58 vars in 1015 bytes} [Sun Aug 21 01:18:24 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:24.335332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b403cb60-86b7-4a8d-9110-3b8b35b5389d None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:24.335614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b403cb60-86b7-4a8d-9110-3b8b35b5389d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:24.335943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b403cb60-86b7-4a8d-9110-3b8b35b5389d None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:24.337286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 708/1412] 10.222.0.47 () {54 vars in 918 bytes} [Sun Aug 21 01:18:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:18:24.342192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6111d3f3-d55e-4285-b27c-bd4fc91fa968 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:24.342473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6111d3f3-d55e-4285-b27c-bd4fc91fa968 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:24.342743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6111d3f3-d55e-4285-b27c-bd4fc91fa968 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:24.415598 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:24.418434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-6111d3f3-d55e-4285-b27c-bd4fc91fa968 None None] MFA Rules not processed for user `fabdcdc25d1d4ef6bc66bd6cadecaaf8`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:24.419628 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6111d3f3-d55e-4285-b27c-bd4fc91fa968 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:24.537582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 705/1413] 10.222.0.47 () {58 vars in 1015 bytes} [Sun Aug 21 01:18:24 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:26.942398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6007e1e9-91b2-40a0-96ba-9e5acb0ed44d None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:26.942970 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6007e1e9-91b2-40a0-96ba-9e5acb0ed44d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:26.942970 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6007e1e9-91b2-40a0-96ba-9e5acb0ed44d None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:26.944344 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 709/1414] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 21 01:18:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:18:26.949867 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f088b7d-dcb9-4215-bd72-d5cebd19e2a6 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:26.950292 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f088b7d-dcb9-4215-bd72-d5cebd19e2a6 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:26.950388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f088b7d-dcb9-4215-bd72-d5cebd19e2a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:27.017357 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-6f088b7d-dcb9-4215-bd72-d5cebd19e2a6 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:27.018593 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6f088b7d-dcb9-4215-bd72-d5cebd19e2a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:27.075397 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:27.118575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 706/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 21 01:18:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:27.430134 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6007e1e9-91b2-40a0-96ba-9e5acb0ed44d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:27.431715 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-6007e1e9-91b2-40a0-96ba-9e5acb0ed44d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:27.578813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6007e1e9-91b2-40a0-96ba-9e5acb0ed44d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:27.655925 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:27.696871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba6fae7-42e3-46f8-8a19-9f6eeecaeccd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:27.697475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba6fae7-42e3-46f8-8a19-9f6eeecaeccd service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:27.697753 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba6fae7-42e3-46f8-8a19-9f6eeecaeccd service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:27.698021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba6fae7-42e3-46f8-8a19-9f6eeecaeccd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:27.699618 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7ba6fae7-42e3-46f8-8a19-9f6eeecaeccd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:27.790351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba6fae7-42e3-46f8-8a19-9f6eeecaeccd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:27.791935 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba6fae7-42e3-46f8-8a19-9f6eeecaeccd service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:27.810106 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 710/1416] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:28.713340 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17c14d5-3c59-44cc-b200-0d153448e2a3 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:28.713883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17c14d5-3c59-44cc-b200-0d153448e2a3 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:28.713883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17c14d5-3c59-44cc-b200-0d153448e2a3 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:28.715231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 707/1417] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 21 01:18:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:28.720268 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472a693d-1951-4308-ae5f-1e0de39dcc3d None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:28.720754 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472a693d-1951-4308-ae5f-1e0de39dcc3d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:28.720858 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472a693d-1951-4308-ae5f-1e0de39dcc3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:28.796945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-472a693d-1951-4308-ae5f-1e0de39dcc3d None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:28.798142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-472a693d-1951-4308-ae5f-1e0de39dcc3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:28.825295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:28.896981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 711/1418] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 21 01:18:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:29.060492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17c14d5-3c59-44cc-b200-0d153448e2a3 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:29.061939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-f17c14d5-3c59-44cc-b200-0d153448e2a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:29.209360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17c14d5-3c59-44cc-b200-0d153448e2a3 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:29.248711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:29.324341 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843b9fbd-6628-4d9a-8ce6-bb9bc881f83a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:29.324862 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843b9fbd-6628-4d9a-8ce6-bb9bc881f83a service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:29.325133 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843b9fbd-6628-4d9a-8ce6-bb9bc881f83a service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:29.325393 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843b9fbd-6628-4d9a-8ce6-bb9bc881f83a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:29.326979 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-843b9fbd-6628-4d9a-8ce6-bb9bc881f83a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:29.417336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843b9fbd-6628-4d9a-8ce6-bb9bc881f83a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:29.418939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843b9fbd-6628-4d9a-8ce6-bb9bc881f83a service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:29.437624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 708/1419] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:30.336369 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97611f1a-d0a6-4563-a40c-e03a5ed7e9de None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:30.336971 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97611f1a-d0a6-4563-a40c-e03a5ed7e9de None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:30.336971 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97611f1a-d0a6-4563-a40c-e03a5ed7e9de None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:30.338276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 712/1420] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 21 01:18:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:30.343546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06222dec-c960-42aa-80bd-85522837a713 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:30.344434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06222dec-c960-42aa-80bd-85522837a713 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:30.344708 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06222dec-c960-42aa-80bd-85522837a713 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:30.402291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:30.427274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-06222dec-c960-42aa-80bd-85522837a713 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:30.428647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-06222dec-c960-42aa-80bd-85522837a713 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:30.526295 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 709/1421] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 21 01:18:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:30.680824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97611f1a-d0a6-4563-a40c-e03a5ed7e9de None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:30.682386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-97611f1a-d0a6-4563-a40c-e03a5ed7e9de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:30.832702 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97611f1a-d0a6-4563-a40c-e03a5ed7e9de None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:30.844913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:30.949346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2288d3f0-16bb-482d-808d-605b58c715fe service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:30.949953 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2288d3f0-16bb-482d-808d-605b58c715fe service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:30.950232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2288d3f0-16bb-482d-808d-605b58c715fe service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:30.950502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2288d3f0-16bb-482d-808d-605b58c715fe service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:30.952128 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-2288d3f0-16bb-482d-808d-605b58c715fe service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:31.050312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2288d3f0-16bb-482d-808d-605b58c715fe service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:31.051904 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2288d3f0-16bb-482d-808d-605b58c715fe service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:31.072350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 713/1422] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:31.951811 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37458bf3-9d14-4672-9373-92e2ff0ecac2 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:31.952400 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37458bf3-9d14-4672-9373-92e2ff0ecac2 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:31.952400 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37458bf3-9d14-4672-9373-92e2ff0ecac2 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:31.953745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 710/1423] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 21 01:18:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:31.959301 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4de143a-129d-48e2-a7f1-bf856a0fad5e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:31.959721 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4de143a-129d-48e2-a7f1-bf856a0fad5e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:31.959837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4de143a-129d-48e2-a7f1-bf856a0fad5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:31.980148 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:32.037414 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-d4de143a-129d-48e2-a7f1-bf856a0fad5e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:32.038623 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d4de143a-129d-48e2-a7f1-bf856a0fad5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:32.138393 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 714/1424] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 21 01:18:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:32.298016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37458bf3-9d14-4672-9373-92e2ff0ecac2 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:32.300274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-37458bf3-9d14-4672-9373-92e2ff0ecac2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:32.414266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:32.452313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37458bf3-9d14-4672-9373-92e2ff0ecac2 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:32.571282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc4b538-0385-41c6-ba90-9045b05cd772 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:32.571803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc4b538-0385-41c6-ba90-9045b05cd772 service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:32.572077 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc4b538-0385-41c6-ba90-9045b05cd772 service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:32.572379 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc4b538-0385-41c6-ba90-9045b05cd772 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:32.574000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ebc4b538-0385-41c6-ba90-9045b05cd772 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:32.655893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:32.669553 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc4b538-0385-41c6-ba90-9045b05cd772 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:32.671220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc4b538-0385-41c6-ba90-9045b05cd772 service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:32.691326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 711/1425] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:33.565278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee1ab52-d202-4fd4-9236-8d85a7b30e5b None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:33.566402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee1ab52-d202-4fd4-9236-8d85a7b30e5b None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:33.566402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee1ab52-d202-4fd4-9236-8d85a7b30e5b None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:33.567463 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 715/1426] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 21 01:18:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:33.572994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6848ecea-1025-45d0-adc5-fa08afbea73b None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:33.573526 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6848ecea-1025-45d0-adc5-fa08afbea73b None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:33.573831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6848ecea-1025-45d0-adc5-fa08afbea73b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:33.660765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-6848ecea-1025-45d0-adc5-fa08afbea73b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:33.661864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6848ecea-1025-45d0-adc5-fa08afbea73b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:33.759055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 712/1427] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 21 01:18:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:33.910275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee1ab52-d202-4fd4-9236-8d85a7b30e5b None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:33.911865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5ee1ab52-d202-4fd4-9236-8d85a7b30e5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:33.997522 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:34.060557 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee1ab52-d202-4fd4-9236-8d85a7b30e5b None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:34.177682 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2138d3da-0121-4110-9565-517c9ee47cba service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:34.177682 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2138d3da-0121-4110-9565-517c9ee47cba service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:34.177682 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2138d3da-0121-4110-9565-517c9ee47cba service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:34.177682 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2138d3da-0121-4110-9565-517c9ee47cba service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:34.178194 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-2138d3da-0121-4110-9565-517c9ee47cba service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:34.228707 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:34.270342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2138d3da-0121-4110-9565-517c9ee47cba service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:34.271928 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2138d3da-0121-4110-9565-517c9ee47cba service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:34.290453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 716/1428] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:35.168468 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22a1dc6-b642-48a3-a24d-8ad514793f4b None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:35.169002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22a1dc6-b642-48a3-a24d-8ad514793f4b None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:35.169002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22a1dc6-b642-48a3-a24d-8ad514793f4b None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:35.170534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 713/1429] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 21 01:18:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:35.175594 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4910bee-5f6d-427f-a5c6-9279406aebe8 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:35.176013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4910bee-5f6d-427f-a5c6-9279406aebe8 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:35.180429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4910bee-5f6d-427f-a5c6-9279406aebe8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:35.253103 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-b4910bee-5f6d-427f-a5c6-9279406aebe8 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:35.254243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b4910bee-5f6d-427f-a5c6-9279406aebe8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:35.351973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 717/1430] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 21 01:18:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:35.504461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22a1dc6-b642-48a3-a24d-8ad514793f4b None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:35.506044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-a22a1dc6-b642-48a3-a24d-8ad514793f4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:35.561543 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:35.663931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22a1dc6-b642-48a3-a24d-8ad514793f4b None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:35.778528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08603fa6-52df-4770-90d3-6b1e7909164a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:35.779049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08603fa6-52df-4770-90d3-6b1e7909164a service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:35.779323 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08603fa6-52df-4770-90d3-6b1e7909164a service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:35.779597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08603fa6-52df-4770-90d3-6b1e7909164a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:35.781362 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-08603fa6-52df-4770-90d3-6b1e7909164a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:35.789158 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:35.874328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08603fa6-52df-4770-90d3-6b1e7909164a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:35.875944 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08603fa6-52df-4770-90d3-6b1e7909164a service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:35.895833 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 714/1431] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:36.767199 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d4a85c-8095-444d-90ef-ba4506db8ebb None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:36.767743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d4a85c-8095-444d-90ef-ba4506db8ebb None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:36.767743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d4a85c-8095-444d-90ef-ba4506db8ebb None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:36.769200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 718/1432] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 21 01:18:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:36.774194 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812d2939-088a-4b3d-8228-681f10e70bc1 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:36.774582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812d2939-088a-4b3d-8228-681f10e70bc1 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:36.774735 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812d2939-088a-4b3d-8228-681f10e70bc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:36.854668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-812d2939-088a-4b3d-8228-681f10e70bc1 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:36.855880 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-812d2939-088a-4b3d-8228-681f10e70bc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:36.930121 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:36.963735 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 715/1433] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 21 01:18:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:37.119026 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d4a85c-8095-444d-90ef-ba4506db8ebb None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:37.120494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a5d4a85c-8095-444d-90ef-ba4506db8ebb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:37.138699 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:37.276473 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d4a85c-8095-444d-90ef-ba4506db8ebb None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:37.372884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:37.400089 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3faa6a92-dcf4-43a3-bbd4-85df5cabea57 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:37.400733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faa6a92-dcf4-43a3-bbd4-85df5cabea57 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:37.401005 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faa6a92-dcf4-43a3-bbd4-85df5cabea57 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:37.401269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faa6a92-dcf4-43a3-bbd4-85df5cabea57 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:37.402861 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-3faa6a92-dcf4-43a3-bbd4-85df5cabea57 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:37.494794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3faa6a92-dcf4-43a3-bbd4-85df5cabea57 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:37.496400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3faa6a92-dcf4-43a3-bbd4-85df5cabea57 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:37.514819 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 719/1434] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:38.384865 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821bd2fc-ef49-404a-a3ff-3a9ab1187742 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:38.385385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821bd2fc-ef49-404a-a3ff-3a9ab1187742 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:38.385385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821bd2fc-ef49-404a-a3ff-3a9ab1187742 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:38.386764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 716/1435] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 21 01:18:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:38.391847 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16038fc-42ee-4012-b446-f8a590376433 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:38.392423 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16038fc-42ee-4012-b446-f8a590376433 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:38.392594 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16038fc-42ee-4012-b446-f8a590376433 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:38.468555 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-e16038fc-42ee-4012-b446-f8a590376433 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:38.469755 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e16038fc-42ee-4012-b446-f8a590376433 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:38.510609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:38.566598 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 720/1436] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 21 01:18:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:38.718945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821bd2fc-ef49-404a-a3ff-3a9ab1187742 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:38.720494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-821bd2fc-ef49-404a-a3ff-3a9ab1187742 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:38.866994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821bd2fc-ef49-404a-a3ff-3a9ab1187742 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:38.923337 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:38.983625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61210c4d-bb83-4e2c-9e0a-b4ff2b33b3a2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:38.984191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61210c4d-bb83-4e2c-9e0a-b4ff2b33b3a2 service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:38.984469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61210c4d-bb83-4e2c-9e0a-b4ff2b33b3a2 service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:38.984737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61210c4d-bb83-4e2c-9e0a-b4ff2b33b3a2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:38.986360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-61210c4d-bb83-4e2c-9e0a-b4ff2b33b3a2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:39.078200 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61210c4d-bb83-4e2c-9e0a-b4ff2b33b3a2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:39.079824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61210c4d-bb83-4e2c-9e0a-b4ff2b33b3a2 service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:39.097891 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 717/1437] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:39.943220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cc60be-6e0b-4763-b1c5-253098663e82 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:39.943786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cc60be-6e0b-4763-b1c5-253098663e82 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:39.943786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cc60be-6e0b-4763-b1c5-253098663e82 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:39.945180 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 721/1438] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 21 01:18:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:39.950529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1918dfb9-d3b9-4851-b668-1c410b9ccfed None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:39.950939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1918dfb9-d3b9-4851-b668-1c410b9ccfed None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:39.951080 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1918dfb9-d3b9-4851-b668-1c410b9ccfed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:40.037452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-1918dfb9-d3b9-4851-b668-1c410b9ccfed None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:40.038800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1918dfb9-d3b9-4851-b668-1c410b9ccfed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:40.040945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:40.138356 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 718/1439] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 21 01:18:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:40.287303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cc60be-6e0b-4763-b1c5-253098663e82 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:40.288891 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-61cc60be-6e0b-4763-b1c5-253098663e82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:40.438343 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cc60be-6e0b-4763-b1c5-253098663e82 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:40.466951 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:40.556751 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7eeebf-9932-456a-bd0c-95cf7eea5ca7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:40.557275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7eeebf-9932-456a-bd0c-95cf7eea5ca7 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:40.557551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7eeebf-9932-456a-bd0c-95cf7eea5ca7 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:40.557817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7eeebf-9932-456a-bd0c-95cf7eea5ca7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:40.559703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-0f7eeebf-9932-456a-bd0c-95cf7eea5ca7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:40.648993 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7eeebf-9932-456a-bd0c-95cf7eea5ca7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:40.650679 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7eeebf-9932-456a-bd0c-95cf7eea5ca7 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:40.668572 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 722/1440] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:42.489302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce71307-c442-4da1-8a31-dc3b6627a281 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:42.489841 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce71307-c442-4da1-8a31-dc3b6627a281 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:42.489841 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce71307-c442-4da1-8a31-dc3b6627a281 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:42.491111 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 719/1441] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 21 01:18:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:42.496419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1810529d-d5ea-4fc1-ab1f-f07a861a6402 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:42.496861 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1810529d-d5ea-4fc1-ab1f-f07a861a6402 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:42.496959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1810529d-d5ea-4fc1-ab1f-f07a861a6402 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:42.537702 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:42.563189 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-1810529d-d5ea-4fc1-ab1f-f07a861a6402 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:42.564441 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-1810529d-d5ea-4fc1-ab1f-f07a861a6402 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:42.661174 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 723/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 21 01:18:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:42.946608 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce71307-c442-4da1-8a31-dc3b6627a281 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:42.948175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-bce71307-c442-4da1-8a31-dc3b6627a281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:43.055583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce71307-c442-4da1-8a31-dc3b6627a281 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:43.087744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:43.128685 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6749639-e33e-45db-ac8c-0b4c9486421e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:43.129293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6749639-e33e-45db-ac8c-0b4c9486421e service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:43.129566 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6749639-e33e-45db-ac8c-0b4c9486421e service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:43.129829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6749639-e33e-45db-ac8c-0b4c9486421e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:43.131427 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e6749639-e33e-45db-ac8c-0b4c9486421e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:43.239943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6749639-e33e-45db-ac8c-0b4c9486421e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:43.241551 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6749639-e33e-45db-ac8c-0b4c9486421e service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:43.260000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 720/1443] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:18:45.346012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd16de7b-edd3-49ba-ae36-1533be2866e2 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:45.347254 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd16de7b-edd3-49ba-ae36-1533be2866e2 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:45.347254 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd16de7b-edd3-49ba-ae36-1533be2866e2 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:45.348453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 724/1444] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 21 01:18:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:45.353878 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf2777d-9a97-4861-bd6a-f107cc51fb3b None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:45.354302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf2777d-9a97-4861-bd6a-f107cc51fb3b None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:45.354404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf2777d-9a97-4861-bd6a-f107cc51fb3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:45.421971 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-cdf2777d-9a97-4861-bd6a-f107cc51fb3b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:45.423237 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-cdf2777d-9a97-4861-bd6a-f107cc51fb3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:45.436568 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:45.521141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 721/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 21 01:18:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:45.802835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd16de7b-edd3-49ba-ae36-1533be2866e2 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:45.804505 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-bd16de7b-edd3-49ba-ae36-1533be2866e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:45.909569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd16de7b-edd3-49ba-ae36-1533be2866e2 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:45.921658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:45.984052 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e91887-43ca-4b9a-bad7-152da6474bdd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:45.984680 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e91887-43ca-4b9a-bad7-152da6474bdd service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:45.984967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e91887-43ca-4b9a-bad7-152da6474bdd service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:45.985230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e91887-43ca-4b9a-bad7-152da6474bdd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:45.986808 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b0e91887-43ca-4b9a-bad7-152da6474bdd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:46.097725 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e91887-43ca-4b9a-bad7-152da6474bdd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:46.099069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e91887-43ca-4b9a-bad7-152da6474bdd service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:46.117914 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 725/1446] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:48.463513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d47e68-1daf-47d7-9314-a713ece9f1b3 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:48.464085 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d47e68-1daf-47d7-9314-a713ece9f1b3 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:48.464085 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d47e68-1daf-47d7-9314-a713ece9f1b3 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:48.465432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 722/1447] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 21 01:18:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:18:48.471335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3f642f-8672-42bd-9eac-b214be135f8f None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:48.471890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3f642f-8672-42bd-9eac-b214be135f8f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:48.471890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3f642f-8672-42bd-9eac-b214be135f8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:48.524647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:48.543661 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-ca3f642f-8672-42bd-9eac-b214be135f8f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:48.544928 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ca3f642f-8672-42bd-9eac-b214be135f8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:48.651495 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 726/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 21 01:18:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:48.954330 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d47e68-1daf-47d7-9314-a713ece9f1b3 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:48.954330 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-f3d47e68-1daf-47d7-9314-a713ece9f1b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:49.069395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d47e68-1daf-47d7-9314-a713ece9f1b3 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:49.113508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:49.150842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2a35a0-39c4-4694-b37a-a3f48f9d13fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:49.151444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2a35a0-39c4-4694-b37a-a3f48f9d13fb service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:49.151717 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2a35a0-39c4-4694-b37a-a3f48f9d13fb service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:49.151980 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2a35a0-39c4-4694-b37a-a3f48f9d13fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:49.153619 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-0a2a35a0-39c4-4694-b37a-a3f48f9d13fb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:49.270327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2a35a0-39c4-4694-b37a-a3f48f9d13fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:49.271799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2a35a0-39c4-4694-b37a-a3f48f9d13fb service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:49.291333 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 723/1449] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:51.363872 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b5398-9790-4671-8e59-686213710a76 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:51.364581 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b5398-9790-4671-8e59-686213710a76 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:51.364581 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b5398-9790-4671-8e59-686213710a76 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:51.365834 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 727/1450] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 21 01:18:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:51.371420 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a0a0ed-b49e-426a-b904-19192b17c10c None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:51.372052 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a0a0ed-b49e-426a-b904-19192b17c10c None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:51.372052 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a0a0ed-b49e-426a-b904-19192b17c10c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:51.444529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-75a0a0ed-b49e-426a-b904-19192b17c10c None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:51.445767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-75a0a0ed-b49e-426a-b904-19192b17c10c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:51.467818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:51.552494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 724/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 21 01:18:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:51.838029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891b5398-9790-4671-8e59-686213710a76 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:51.839611 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-891b5398-9790-4671-8e59-686213710a76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:51.962252 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891b5398-9790-4671-8e59-686213710a76 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:51.986031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:52.044842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ca45df-36eb-4789-afc1-7c128cb64d04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:52.045432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ca45df-36eb-4789-afc1-7c128cb64d04 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:52.045708 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ca45df-36eb-4789-afc1-7c128cb64d04 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:52.045973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ca45df-36eb-4789-afc1-7c128cb64d04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:52.047586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f6ca45df-36eb-4789-afc1-7c128cb64d04 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:52.174624 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ca45df-36eb-4789-afc1-7c128cb64d04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:52.175981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ca45df-36eb-4789-afc1-7c128cb64d04 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:52.196647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 728/1452] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:54.239372 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46820baf-4aad-4c93-a2c5-18bd22f7e0ee None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:54.239951 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46820baf-4aad-4c93-a2c5-18bd22f7e0ee None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:54.239951 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46820baf-4aad-4c93-a2c5-18bd22f7e0ee None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:54.241298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 725/1453] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 21 01:18:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:54.246771 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827856b3-d60e-4966-a82e-c9585f62c746 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:54.247313 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827856b3-d60e-4966-a82e-c9585f62c746 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:54.247313 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827856b3-d60e-4966-a82e-c9585f62c746 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:54.312184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:54.315688 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-827856b3-d60e-4966-a82e-c9585f62c746 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:54.316953 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-827856b3-d60e-4966-a82e-c9585f62c746 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:54.415339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 729/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 21 01:18:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:54.701473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46820baf-4aad-4c93-a2c5-18bd22f7e0ee None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:54.702427 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-46820baf-4aad-4c93-a2c5-18bd22f7e0ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:54.820754 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46820baf-4aad-4c93-a2c5-18bd22f7e0ee None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:54.872738 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:54.899292 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2931b2-f4d7-4e53-8044-6a43df179329 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:54.899950 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2931b2-f4d7-4e53-8044-6a43df179329 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:54.900255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2931b2-f4d7-4e53-8044-6a43df179329 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:54.900528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2931b2-f4d7-4e53-8044-6a43df179329 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:54.902140 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ee2931b2-f4d7-4e53-8044-6a43df179329 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:55.018513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2931b2-f4d7-4e53-8044-6a43df179329 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:55.019976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2931b2-f4d7-4e53-8044-6a43df179329 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:55.039198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 726/1455] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:57.009176 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef766c0-cfe3-4805-8028-453c06bab535 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:57.009762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef766c0-cfe3-4805-8028-453c06bab535 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:57.009762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef766c0-cfe3-4805-8028-453c06bab535 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:57.011068 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 730/1456] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 21 01:18:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:57.016638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4409c4a9-55b9-49ef-977d-34cda1f7da9e None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:57.017080 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4409c4a9-55b9-49ef-977d-34cda1f7da9e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:57.017209 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4409c4a9-55b9-49ef-977d-34cda1f7da9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:57.085363 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-4409c4a9-55b9-49ef-977d-34cda1f7da9e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:57.086585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-4409c4a9-55b9-49ef-977d-34cda1f7da9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:57.117241 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:57.190597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 727/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 21 01:18:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:18:57.475137 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef766c0-cfe3-4805-8028-453c06bab535 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:18:57.476723 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-fef766c0-cfe3-4805-8028-453c06bab535 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:57.581845 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef766c0-cfe3-4805-8028-453c06bab535 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:18:57.610822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:57.656606 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a12e0d6-ef3c-4a65-8989-d73cd364c5d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:18:57.657227 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a12e0d6-ef3c-4a65-8989-d73cd364c5d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:57.657492 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a12e0d6-ef3c-4a65-8989-d73cd364c5d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:57.657763 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a12e0d6-ef3c-4a65-8989-d73cd364c5d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:57.659945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9a12e0d6-ef3c-4a65-8989-d73cd364c5d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:57.773949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a12e0d6-ef3c-4a65-8989-d73cd364c5d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:18:57.775415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a12e0d6-ef3c-4a65-8989-d73cd364c5d4 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:18:57.794738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 731/1458] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:18:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:18:59.753944 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b1e521-abff-496d-b7e1-5d2eba6c5202 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:59.754492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b1e521-abff-496d-b7e1-5d2eba6c5202 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:59.754492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b1e521-abff-496d-b7e1-5d2eba6c5202 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:59.755803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 728/1459] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 21 01:18:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:18:59.761120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e550232-b4bb-414a-9987-3a20ee9bd8e6 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:18:59.761554 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e550232-b4bb-414a-9987-3a20ee9bd8e6 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:18:59.761669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e550232-b4bb-414a-9987-3a20ee9bd8e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:18:59.829060 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-0e550232-b4bb-414a-9987-3a20ee9bd8e6 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:18:59.830275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-0e550232-b4bb-414a-9987-3a20ee9bd8e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:18:59.838665 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:18:59.932239 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 732/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 21 01:18:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:19:00.215620 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b1e521-abff-496d-b7e1-5d2eba6c5202 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:19:00.217207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-88b1e521-abff-496d-b7e1-5d2eba6c5202 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:00.325519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b1e521-abff-496d-b7e1-5d2eba6c5202 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:19:00.382391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:00.401391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11740542-18b3-48d2-bb2a-13f2fb99f613 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:19:00.401994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11740542-18b3-48d2-bb2a-13f2fb99f613 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:00.402267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11740542-18b3-48d2-bb2a-13f2fb99f613 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:00.402617 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11740542-18b3-48d2-bb2a-13f2fb99f613 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:00.404395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-11740542-18b3-48d2-bb2a-13f2fb99f613 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:00.521405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11740542-18b3-48d2-bb2a-13f2fb99f613 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:19:00.522894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11740542-18b3-48d2-bb2a-13f2fb99f613 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:19:00.540842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 729/1461] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:19:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:19:02.531233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236140fc-f820-4a68-980b-8ca70308cf55 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:02.532003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236140fc-f820-4a68-980b-8ca70308cf55 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:02.532003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236140fc-f820-4a68-980b-8ca70308cf55 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:02.533271 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 733/1462] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 21 01:19:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:19:02.540082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a8754a-3bbc-4e50-9cdc-7bcf00a6d889 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:02.541033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a8754a-3bbc-4e50-9cdc-7bcf00a6d889 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:02.542390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a8754a-3bbc-4e50-9cdc-7bcf00a6d889 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:02.623628 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-13a8754a-3bbc-4e50-9cdc-7bcf00a6d889 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:19:02.624791 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-13a8754a-3bbc-4e50-9cdc-7bcf00a6d889 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:02.669559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:02.740130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 730/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 21 01:19:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:19:03.022293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236140fc-f820-4a68-980b-8ca70308cf55 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:19:03.023847 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-236140fc-f820-4a68-980b-8ca70308cf55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:03.146652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236140fc-f820-4a68-980b-8ca70308cf55 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:19:03.194486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:03.235307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43a3a92-b480-4ff4-b22a-2c2c21626b32 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:19:03.235855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43a3a92-b480-4ff4-b22a-2c2c21626b32 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:03.236207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43a3a92-b480-4ff4-b22a-2c2c21626b32 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:03.236480 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43a3a92-b480-4ff4-b22a-2c2c21626b32 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:03.238117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b43a3a92-b480-4ff4-b22a-2c2c21626b32 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:03.368068 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43a3a92-b480-4ff4-b22a-2c2c21626b32 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:19:03.369539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43a3a92-b480-4ff4-b22a-2c2c21626b32 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:19:03.392751 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 734/1464] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:19:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:19:05.409648 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1285c9-a808-4f6b-9940-719cd39d3684 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:05.410194 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1285c9-a808-4f6b-9940-719cd39d3684 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:05.410194 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1285c9-a808-4f6b-9940-719cd39d3684 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:05.411574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 731/1465] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 21 01:19:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:19:05.417420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc29b1a-fe1c-4b71-ad3e-c0d51bd0d822 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:05.417922 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc29b1a-fe1c-4b71-ad3e-c0d51bd0d822 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:05.418027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc29b1a-fe1c-4b71-ad3e-c0d51bd0d822 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:05.506166 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-afc29b1a-fe1c-4b71-ad3e-c0d51bd0d822 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:19:05.508272 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-afc29b1a-fe1c-4b71-ad3e-c0d51bd0d822 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:05.534719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:05.634564 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 735/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 21 01:19:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:19:05.930980 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1285c9-a808-4f6b-9940-719cd39d3684 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:19:05.932572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-dd1285c9-a808-4f6b-9940-719cd39d3684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:06.058408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1285c9-a808-4f6b-9940-719cd39d3684 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:19:06.139613 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:06.146298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10ed8f3-e192-495c-b81e-5ce39179505e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:19:06.146908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10ed8f3-e192-495c-b81e-5ce39179505e service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:06.147183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10ed8f3-e192-495c-b81e-5ce39179505e service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:06.147453 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10ed8f3-e192-495c-b81e-5ce39179505e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:06.149080 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c10ed8f3-e192-495c-b81e-5ce39179505e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:06.268617 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10ed8f3-e192-495c-b81e-5ce39179505e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:19:06.270087 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10ed8f3-e192-495c-b81e-5ce39179505e service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:19:06.289117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 732/1467] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:19:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:19:08.259940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86501961-a988-4d01-acbb-c1072955aac9 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:08.260611 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86501961-a988-4d01-acbb-c1072955aac9 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:08.260779 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86501961-a988-4d01-acbb-c1072955aac9 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:08.262071 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 736/1468] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 21 01:19:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:19:08.269186 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ba2dbe-8b6f-4014-bc62-c70db03b41c2 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:08.269186 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ba2dbe-8b6f-4014-bc62-c70db03b41c2 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:08.269186 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ba2dbe-8b6f-4014-bc62-c70db03b41c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:08.341701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-31ba2dbe-8b6f-4014-bc62-c70db03b41c2 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:19:08.342933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-31ba2dbe-8b6f-4014-bc62-c70db03b41c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:08.398496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:08.458789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 733/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 21 01:19:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:19:08.746966 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86501961-a988-4d01-acbb-c1072955aac9 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:19:08.748817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-86501961-a988-4d01-acbb-c1072955aac9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:08.865901 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86501961-a988-4d01-acbb-c1072955aac9 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:19:08.914770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:08.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360704a9-f431-4c51-90af-8d598039d728 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:19:08.942111 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360704a9-f431-4c51-90af-8d598039d728 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:08.942381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360704a9-f431-4c51-90af-8d598039d728 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:08.942646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360704a9-f431-4c51-90af-8d598039d728 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:08.944282 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-360704a9-f431-4c51-90af-8d598039d728 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:09.065399 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360704a9-f431-4c51-90af-8d598039d728 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:19:09.066750 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360704a9-f431-4c51-90af-8d598039d728 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:19:09.086745 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 737/1470] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:19:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:19:18.456935 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=22 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:19.084211 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=66 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:28.870421 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5af6e1-a342-4f54-85eb-e132f863cdcf None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:28.871016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5af6e1-a342-4f54-85eb-e132f863cdcf None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:28.871016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5af6e1-a342-4f54-85eb-e132f863cdcf None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:28.872379 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 734/1471] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:19:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:19:28.878328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631fb1e2-bfb1-415b-9997-bef81d4cfd4e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:28.878767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631fb1e2-bfb1-415b-9997-bef81d4cfd4e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:28.878893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631fb1e2-bfb1-415b-9997-bef81d4cfd4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:28.946364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-631fb1e2-bfb1-415b-9997-bef81d4cfd4e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:19:28.947588 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-631fb1e2-bfb1-415b-9997-bef81d4cfd4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:29.043880 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 738/1472] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:19:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:19:29.351963 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54a1743-621e-4112-a0d1-2acc5727ed73 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:29.352271 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54a1743-621e-4112-a0d1-2acc5727ed73 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:29.352546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54a1743-621e-4112-a0d1-2acc5727ed73 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:29.353895 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 735/1473] 10.222.0.47 () {54 vars in 967 bytes} [Sun Aug 21 01:19:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:19:29.358778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9e6df9-e3d9-452e-9cd8-0271a1534a33 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:29.359069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9e6df9-e3d9-452e-9cd8-0271a1534a33 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:29.359346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9e6df9-e3d9-452e-9cd8-0271a1534a33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:29.436865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-9c9e6df9-e3d9-452e-9cd8-0271a1534a33 None None] MFA Rules not processed for user `a6365b2328f34ab9b1b73ececbf5cc30`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:19:29.438090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9c9e6df9-e3d9-452e-9cd8-0271a1534a33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:29.461893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:29.519423 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 739/1474] 10.222.0.47 () {58 vars in 1064 bytes} [Sun Aug 21 01:19:29 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:19:29.527246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54a1743-621e-4112-a0d1-2acc5727ed73 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:19:29.528890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-f54a1743-621e-4112-a0d1-2acc5727ed73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:29.638925 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54a1743-621e-4112-a0d1-2acc5727ed73 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:19:29.715447 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47928346-6788-44ba-afe8-718f5df7784b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6365b2328f34ab9b1b73ececbf5cc30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:19:29.716075 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47928346-6788-44ba-afe8-718f5df7784b service glance] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:29.716428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47928346-6788-44ba-afe8-718f5df7784b service glance] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:29.716687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47928346-6788-44ba-afe8-718f5df7784b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:29.718375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-47928346-6788-44ba-afe8-718f5df7784b service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:29.893584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:29.949141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47928346-6788-44ba-afe8-718f5df7784b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:19:29.950577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47928346-6788-44ba-afe8-718f5df7784b service glance] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:19:29.969873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 736/1475] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:19:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:19:31.978171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff84f81-ef1b-4512-bd25-14fe18afc28f None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:31.978749 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff84f81-ef1b-4512-bd25-14fe18afc28f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:31.978749 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff84f81-ef1b-4512-bd25-14fe18afc28f None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:31.980042 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 740/1476] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:19:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:19:31.986477 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2194c339-d944-4841-9f14-e17ccb2742dd None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:31.988214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2194c339-d944-4841-9f14-e17ccb2742dd None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:31.988214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2194c339-d944-4841-9f14-e17ccb2742dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:32.059475 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-2194c339-d944-4841-9f14-e17ccb2742dd None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:19:32.060730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-2194c339-d944-4841-9f14-e17ccb2742dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:32.170058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 737/1477] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:19:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:19:32.452883 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff84f81-ef1b-4512-bd25-14fe18afc28f None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:19:32.454470 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-dff84f81-ef1b-4512-bd25-14fe18afc28f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:32.581476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff84f81-ef1b-4512-bd25-14fe18afc28f None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:19:32.630471 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:32.687720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bd78fb-e110-4c38-844d-a832a22db74d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:19:32.688361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bd78fb-e110-4c38-844d-a832a22db74d admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:32.688639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bd78fb-e110-4c38-844d-a832a22db74d admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:32.688907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bd78fb-e110-4c38-844d-a832a22db74d admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:32.689534 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52bd78fb-e110-4c38-844d-a832a22db74d admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:19:32.691240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: WARNING py.warnings [None req-52bd78fb-e110-4c38-844d-a832a22db74d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 21 01:19:32.691240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: warnings.warn(msg) Aug 21 01:19:32.691240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: Aug 21 01:19:32.692460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52bd78fb-e110-4c38-844d-a832a22db74d admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:19:32.711898 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 741/1478] 10.222.0.47 () {56 vars in 1108 bytes} [Sun Aug 21 01:19:32 2022] GET /identity/v3/projects => generated 2553 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:19:34.643332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369aa4dc-359d-402f-9610-95009f2a2a46 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:34.643849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369aa4dc-359d-402f-9610-95009f2a2a46 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:34.643849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369aa4dc-359d-402f-9610-95009f2a2a46 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:34.645178 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 738/1479] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:19:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:19:34.651082 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c912f48-c196-443d-b0ab-ac428c0bcc67 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:34.651537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c912f48-c196-443d-b0ab-ac428c0bcc67 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:34.651841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c912f48-c196-443d-b0ab-ac428c0bcc67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:34.737463 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-0c912f48-c196-443d-b0ab-ac428c0bcc67 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:19:34.737463 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-0c912f48-c196-443d-b0ab-ac428c0bcc67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:34.848152 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 742/1480] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:19:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:19:35.139528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369aa4dc-359d-402f-9610-95009f2a2a46 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:19:35.141117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-369aa4dc-359d-402f-9610-95009f2a2a46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:35.151072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:35.297947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369aa4dc-359d-402f-9610-95009f2a2a46 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:19:35.383192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:35.416821 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5475e16b-551b-4fd8-8e16-005c78c9ebd1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:19:35.417422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5475e16b-551b-4fd8-8e16-005c78c9ebd1 service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:35.417698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5475e16b-551b-4fd8-8e16-005c78c9ebd1 service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:35.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5475e16b-551b-4fd8-8e16-005c78c9ebd1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:35.419599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-5475e16b-551b-4fd8-8e16-005c78c9ebd1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:35.511313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5475e16b-551b-4fd8-8e16-005c78c9ebd1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:19:35.512931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5475e16b-551b-4fd8-8e16-005c78c9ebd1 service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:19:35.532528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 739/1481] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:19:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:19:37.650710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b728480-b8eb-46a0-8914-2c58ec9d2db8 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:37.651247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b728480-b8eb-46a0-8914-2c58ec9d2db8 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:37.651247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b728480-b8eb-46a0-8914-2c58ec9d2db8 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:37.652726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 743/1482] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:19:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:19:37.658263 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff46161-5387-4e80-930d-09818ef01c87 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:37.658688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff46161-5387-4e80-930d-09818ef01c87 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:37.658794 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff46161-5387-4e80-930d-09818ef01c87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:37.725781 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-bff46161-5387-4e80-930d-09818ef01c87 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:19:37.726992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-bff46161-5387-4e80-930d-09818ef01c87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:37.774416 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:37.826412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 740/1483] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:19:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:19:38.111778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b728480-b8eb-46a0-8914-2c58ec9d2db8 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:19:38.113506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-1b728480-b8eb-46a0-8914-2c58ec9d2db8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:38.138618 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:38.223101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b728480-b8eb-46a0-8914-2c58ec9d2db8 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:19:38.302154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a85ca7-738a-49be-b5aa-edd614fedc53 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:19:38.302757 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a85ca7-738a-49be-b5aa-edd614fedc53 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:38.303034 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a85ca7-738a-49be-b5aa-edd614fedc53 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:38.303301 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a85ca7-738a-49be-b5aa-edd614fedc53 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:38.304941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-48a85ca7-738a-49be-b5aa-edd614fedc53 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:38.387310 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:38.425069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a85ca7-738a-49be-b5aa-edd614fedc53 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:19:38.426510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a85ca7-738a-49be-b5aa-edd614fedc53 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:19:38.446634 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 744/1484] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:19:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:19:40.446765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6724bbe5-5c3f-4e0f-8703-2ddef7e60b48 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:40.447332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6724bbe5-5c3f-4e0f-8703-2ddef7e60b48 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:40.447332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6724bbe5-5c3f-4e0f-8703-2ddef7e60b48 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:40.448691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 741/1485] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:19:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:19:40.453985 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a57f61-3a2b-41dd-a15a-8ce616195460 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:40.454397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a57f61-3a2b-41dd-a15a-8ce616195460 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:40.454503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a57f61-3a2b-41dd-a15a-8ce616195460 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:40.521225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-b5a57f61-3a2b-41dd-a15a-8ce616195460 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:19:40.522427 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b5a57f61-3a2b-41dd-a15a-8ce616195460 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:40.625974 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 745/1486] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:19:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:19:40.912128 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6724bbe5-5c3f-4e0f-8703-2ddef7e60b48 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:19:40.913715 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6724bbe5-5c3f-4e0f-8703-2ddef7e60b48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:41.018010 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6724bbe5-5c3f-4e0f-8703-2ddef7e60b48 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:19:41.091069 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:41.092915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b52da9-e6b7-4bcb-baee-49f095e47f6e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:19:41.093514 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b52da9-e6b7-4bcb-baee-49f095e47f6e service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:41.093785 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b52da9-e6b7-4bcb-baee-49f095e47f6e service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:41.094050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b52da9-e6b7-4bcb-baee-49f095e47f6e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:41.095680 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-40b52da9-e6b7-4bcb-baee-49f095e47f6e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:41.206720 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b52da9-e6b7-4bcb-baee-49f095e47f6e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:19:41.208230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b52da9-e6b7-4bcb-baee-49f095e47f6e service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:19:41.226245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 742/1487] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:19:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:19:43.230704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9b5794-c977-49a1-9c72-c33bdd5d7c16 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:43.231240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9b5794-c977-49a1-9c72-c33bdd5d7c16 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:43.231486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9b5794-c977-49a1-9c72-c33bdd5d7c16 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:43.232957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 746/1488] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:19:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:19:43.238664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77771935-d446-4f4b-b794-7bd2b6fa3d26 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:43.239126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77771935-d446-4f4b-b794-7bd2b6fa3d26 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:43.239224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77771935-d446-4f4b-b794-7bd2b6fa3d26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:43.309399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-77771935-d446-4f4b-b794-7bd2b6fa3d26 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:19:43.311009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-77771935-d446-4f4b-b794-7bd2b6fa3d26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:43.363784 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:43.408177 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 743/1489] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:19:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:19:43.701647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9b5794-c977-49a1-9c72-c33bdd5d7c16 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:19:43.703302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7f9b5794-c977-49a1-9c72-c33bdd5d7c16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:43.741316 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:43.816899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9b5794-c977-49a1-9c72-c33bdd5d7c16 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:19:43.907027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79911bc4-e404-4a2b-a230-25a8e8cec2a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:19:43.907622 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79911bc4-e404-4a2b-a230-25a8e8cec2a4 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:43.907902 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79911bc4-e404-4a2b-a230-25a8e8cec2a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:43.908287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79911bc4-e404-4a2b-a230-25a8e8cec2a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:43.909903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-79911bc4-e404-4a2b-a230-25a8e8cec2a4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:43.997345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:44.027694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79911bc4-e404-4a2b-a230-25a8e8cec2a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:19:44.029778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79911bc4-e404-4a2b-a230-25a8e8cec2a4 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:19:44.050182 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 747/1490] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:19:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:19:46.389800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5af6ef9-5ce0-4b38-9045-e93914b01e2a None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:46.390410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5af6ef9-5ce0-4b38-9045-e93914b01e2a None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:46.390410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5af6ef9-5ce0-4b38-9045-e93914b01e2a None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:46.391676 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 744/1491] 10.222.0.47 () {54 vars in 927 bytes} [Sun Aug 21 01:19:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:19:46.397103 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de8cf79-05f1-4fec-8b58-f958a0aeea49 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:46.397565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de8cf79-05f1-4fec-8b58-f958a0aeea49 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:46.397678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de8cf79-05f1-4fec-8b58-f958a0aeea49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:46.469682 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-1de8cf79-05f1-4fec-8b58-f958a0aeea49 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:19:46.470899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-1de8cf79-05f1-4fec-8b58-f958a0aeea49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:46.571535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 748/1492] 10.222.0.47 () {58 vars in 1024 bytes} [Sun Aug 21 01:19:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:19:46.863738 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5af6ef9-5ce0-4b38-9045-e93914b01e2a None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:19:46.865579 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-f5af6ef9-5ce0-4b38-9045-e93914b01e2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:46.973327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5af6ef9-5ce0-4b38-9045-e93914b01e2a None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:19:47.051100 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c65fd6-4f33-4d59-9d20-d87e763a09f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:19:47.052006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c65fd6-4f33-4d59-9d20-d87e763a09f3 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:19:47.052006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c65fd6-4f33-4d59-9d20-d87e763a09f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:19:47.052252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c65fd6-4f33-4d59-9d20-d87e763a09f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:19:47.054095 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-26c65fd6-4f33-4d59-9d20-d87e763a09f3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:19:47.064705 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:47.168986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c65fd6-4f33-4d59-9d20-d87e763a09f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:19:47.170942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c65fd6-4f33-4d59-9d20-d87e763a09f3 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:19:47.191947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 745/1493] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:19:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:19:56.569578 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=82 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:19:57.192198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=50 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:21:32.651397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2713bae9-451d-49f9-bda3-1673963607e4 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:32.651967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2713bae9-451d-49f9-bda3-1673963607e4 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:32.651967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2713bae9-451d-49f9-bda3-1673963607e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:32.756942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-2713bae9-451d-49f9-bda3-1673963607e4 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:21:32.758166 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-2713bae9-451d-49f9-bda3-1673963607e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:21:32.879855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 749/1494] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:21:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:21:32.888073 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c2ff8f-f82a-4c7b-bf3b-076480a1a77a None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:32.888663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c2ff8f-f82a-4c7b-bf3b-076480a1a77a None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:32.888663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c2ff8f-f82a-4c7b-bf3b-076480a1a77a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:32.908771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-81c2ff8f-f82a-4c7b-bf3b-076480a1a77a None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:21:32.909966 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-81c2ff8f-f82a-4c7b-bf3b-076480a1a77a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:21:32.918861 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 746/1495] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:21:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:21:32.923866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2713bae9-451d-49f9-bda3-1673963607e4 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:21:32.935144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2713bae9-451d-49f9-bda3-1673963607e4 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:21:32.939891 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322a537d-3830-411a-9f99-db30423a7db1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:21:32.940535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322a537d-3830-411a-9f99-db30423a7db1 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:32.940818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322a537d-3830-411a-9f99-db30423a7db1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:32.941087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322a537d-3830-411a-9f99-db30423a7db1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:32.942298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322a537d-3830-411a-9f99-db30423a7db1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:21:32.942598 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322a537d-3830-411a-9f99-db30423a7db1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:21:32.943604 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: WARNING py.warnings [None req-322a537d-3830-411a-9f99-db30423a7db1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 21 01:21:32.943604 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: warnings.warn(msg) Aug 21 01:21:32.943604 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: Aug 21 01:21:32.944532 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322a537d-3830-411a-9f99-db30423a7db1 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:21:32.957339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 750/1496] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:21:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:21:32.965654 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c2ff8f-f82a-4c7b-bf3b-076480a1a77a None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:21:32.977221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c2ff8f-f82a-4c7b-bf3b-076480a1a77a None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:21:32.981797 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddea0b65-cec2-490e-9eeb-2c1d91124666 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:21:32.982390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddea0b65-cec2-490e-9eeb-2c1d91124666 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:32.982665 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddea0b65-cec2-490e-9eeb-2c1d91124666 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:32.982925 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddea0b65-cec2-490e-9eeb-2c1d91124666 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:32.983542 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddea0b65-cec2-490e-9eeb-2c1d91124666 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:21:32.985323 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: WARNING py.warnings [None req-ddea0b65-cec2-490e-9eeb-2c1d91124666 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 21 01:21:32.985323 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: warnings.warn(msg) Aug 21 01:21:32.985323 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: Aug 21 01:21:32.986504 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddea0b65-cec2-490e-9eeb-2c1d91124666 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:21:33.105820 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 747/1497] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:21:32 2022] POST /identity/v3/projects => generated 379 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:21:33.110534 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322a537d-3830-411a-9f99-db30423a7db1 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:21:33.131182 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322a537d-3830-411a-9f99-db30423a7db1 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:21:33.190056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:21:33.197242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd54814b-7fc0-4103-8e50-3745fdf700e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:21:33.197917 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd54814b-7fc0-4103-8e50-3745fdf700e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:33.198260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd54814b-7fc0-4103-8e50-3745fdf700e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:33.198445 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd54814b-7fc0-4103-8e50-3745fdf700e5 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:33.199085 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd54814b-7fc0-4103-8e50-3745fdf700e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:21:33.200114 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: WARNING py.warnings [None req-bd54814b-7fc0-4103-8e50-3745fdf700e5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 21 01:21:33.200114 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: warnings.warn(msg) Aug 21 01:21:33.200114 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: Aug 21 01:21:33.201343 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd54814b-7fc0-4103-8e50-3745fdf700e5 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:21:33.372669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 751/1498] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:21:33 2022] POST /identity/v3/users => generated 290 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:21:33.377386 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddea0b65-cec2-490e-9eeb-2c1d91124666 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:21:33.388002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddea0b65-cec2-490e-9eeb-2c1d91124666 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:21:33.392525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fcc268-05af-4a9a-a270-c0629cc231d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:21:33.393180 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fcc268-05af-4a9a-a270-c0629cc231d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:33.393447 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fcc268-05af-4a9a-a270-c0629cc231d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:33.393711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fcc268-05af-4a9a-a270-c0629cc231d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:33.394354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fcc268-05af-4a9a-a270-c0629cc231d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:21:33.395873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: WARNING py.warnings [None req-34fcc268-05af-4a9a-a270-c0629cc231d3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 21 01:21:33.395873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: warnings.warn(msg) Aug 21 01:21:33.395873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: Aug 21 01:21:33.396761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fcc268-05af-4a9a-a270-c0629cc231d3 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:21:33.411506 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 748/1499] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:21:33 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:21:33.418838 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd54814b-7fc0-4103-8e50-3745fdf700e5 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:21:33.429832 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd54814b-7fc0-4103-8e50-3745fdf700e5 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:21:33.434558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854b3798-704e-48b7-a078-379cdc4afb4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:21:33.435159 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854b3798-704e-48b7-a078-379cdc4afb4e admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:33.435434 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854b3798-704e-48b7-a078-379cdc4afb4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:33.435733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854b3798-704e-48b7-a078-379cdc4afb4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:33.436406 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854b3798-704e-48b7-a078-379cdc4afb4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:21:33.437424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: WARNING py.warnings [None req-854b3798-704e-48b7-a078-379cdc4afb4e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 21 01:21:33.437424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: warnings.warn(msg) Aug 21 01:21:33.437424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: Aug 21 01:21:33.438355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854b3798-704e-48b7-a078-379cdc4afb4e admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:21:33.451117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 752/1500] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:21:33 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:21:33.455782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fcc268-05af-4a9a-a270-c0629cc231d3 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:21:33.466163 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fcc268-05af-4a9a-a270-c0629cc231d3 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:21:33.470669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7292358c-8f91-427e-9c64-9760ca9675ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:21:33.471312 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7292358c-8f91-427e-9c64-9760ca9675ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:33.471584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7292358c-8f91-427e-9c64-9760ca9675ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:33.471852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7292358c-8f91-427e-9c64-9760ca9675ae admin admin] PATH_INFO: `/v3/projects/8e9618149b0b46808c35bf93ff4a0b19/users/c09c30563dc94e429980938bf6ab5688/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:33.489846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7292358c-8f91-427e-9c64-9760ca9675ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e9618149b0b46808c35bf93ff4a0b19, user_id=c09c30563dc94e429980938bf6ab5688, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:21:33.490873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: WARNING py.warnings [None req-7292358c-8f91-427e-9c64-9760ca9675ae admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 21 01:21:33.490873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: warnings.warn(msg) Aug 21 01:21:33.490873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: Aug 21 01:21:33.492817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7292358c-8f91-427e-9c64-9760ca9675ae admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:21:33.577547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 749/1501] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:21:33 2022] PUT /identity/v3/projects/8e9618149b0b46808c35bf93ff4a0b19/users/c09c30563dc94e429980938bf6ab5688/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:21:33.594833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8426a152-0e4d-4913-b877-15812a53907e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:33.595113 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8426a152-0e4d-4913-b877-15812a53907e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:33.595455 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8426a152-0e4d-4913-b877-15812a53907e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:33.627275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-8426a152-0e4d-4913-b877-15812a53907e None None] MFA Rules not processed for user `c09c30563dc94e429980938bf6ab5688`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:21:33.632727 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8426a152-0e4d-4913-b877-15812a53907e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:21:33.705650 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 753/1502] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:21:33 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:21:33.724615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d341d78b-d424-4325-8b66-c8cdf8db3894 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:33.724904 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d341d78b-d424-4325-8b66-c8cdf8db3894 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:33.725170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d341d78b-d424-4325-8b66-c8cdf8db3894 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:33.726560 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 750/1503] 10.222.0.47 () {54 vars in 964 bytes} [Sun Aug 21 01:21:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:21:33.732552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d6f3ce-8ffc-45c7-aaee-3397c0e8bdc5 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:33.733006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d6f3ce-8ffc-45c7-aaee-3397c0e8bdc5 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:33.738420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d6f3ce-8ffc-45c7-aaee-3397c0e8bdc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:33.802734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-b6d6f3ce-8ffc-45c7-aaee-3397c0e8bdc5 None None] MFA Rules not processed for user `b953036b8b774f25bda59cec96b21c27`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:21:33.804126 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b6d6f3ce-8ffc-45c7-aaee-3397c0e8bdc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:21:33.896920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 754/1504] 10.222.0.47 () {58 vars in 1061 bytes} [Sun Aug 21 01:21:33 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:21:33.905594 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d341d78b-d424-4325-8b66-c8cdf8db3894 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:21:33.916582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d341d78b-d424-4325-8b66-c8cdf8db3894 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:21:33.920219 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdc76db-9b4d-41ef-80a1-e5c3c7eedb39 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:21:33.920822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdc76db-9b4d-41ef-80a1-e5c3c7eedb39 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:33.921099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdc76db-9b4d-41ef-80a1-e5c3c7eedb39 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:33.921365 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdc76db-9b4d-41ef-80a1-e5c3c7eedb39 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:33.930819 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdc76db-9b4d-41ef-80a1-e5c3c7eedb39 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:21:33.932276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdc76db-9b4d-41ef-80a1-e5c3c7eedb39 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:21:33.942220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 751/1505] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:21:33 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:21:33.993284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489731ee-4a39-4bd2-a795-63841d1f225d None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:33.993569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489731ee-4a39-4bd2-a795-63841d1f225d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:33.993838 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489731ee-4a39-4bd2-a795-63841d1f225d None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:33.995138 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 755/1506] 10.222.0.47 () {52 vars in 830 bytes} [Sun Aug 21 01:21:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:21:34.000506 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdc76db-9b4d-41ef-80a1-e5c3c7eedb39 service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:21:34.070113 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdc76db-9b4d-41ef-80a1-e5c3c7eedb39 service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:21:34.074928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd87e12a-f49a-4624-a288-809a27c06365 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:21:34.075574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd87e12a-f49a-4624-a288-809a27c06365 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:34.075865 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd87e12a-f49a-4624-a288-809a27c06365 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:34.076154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd87e12a-f49a-4624-a288-809a27c06365 admin admin] PATH_INFO: `/v3/users/c09c30563dc94e429980938bf6ab5688` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:34.077349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd87e12a-f49a-4624-a288-809a27c06365 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c09c30563dc94e429980938bf6ab5688)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:21:34.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: WARNING py.warnings [None req-bd87e12a-f49a-4624-a288-809a27c06365 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 21 01:21:34.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: warnings.warn(msg) Aug 21 01:21:34.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: Aug 21 01:21:34.079586 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd87e12a-f49a-4624-a288-809a27c06365 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:21:34.322513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-bd87e12a-f49a-4624-a288-809a27c06365 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c09c30563dc94e429980938bf6ab5688'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:21:34.336415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-bd87e12a-f49a-4624-a288-809a27c06365 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c09c30563dc94e429980938bf6ab5688'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:21:34.363870 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-bd87e12a-f49a-4624-a288-809a27c06365 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c09c30563dc94e429980938bf6ab5688'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:21:34.368462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=88,keystone:INSERT=3,keystone:DELETE=9 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:21:34.462291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-bd87e12a-f49a-4624-a288-809a27c06365 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c09c30563dc94e429980938bf6ab5688'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:21:34.463050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-bd87e12a-f49a-4624-a288-809a27c06365 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c09c30563dc94e429980938bf6ab5688'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:21:34.469742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 752/1507] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:21:33 2022] DELETE /identity/v3/users/c09c30563dc94e429980938bf6ab5688 => generated 0 bytes in 470 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:21:34.474350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef51119d-b157-4465-a1be-41cb5e5ca063 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:34.474623 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef51119d-b157-4465-a1be-41cb5e5ca063 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:34.474890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef51119d-b157-4465-a1be-41cb5e5ca063 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:34.498700 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-ef51119d-b157-4465-a1be-41cb5e5ca063 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:21:34.499897 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ef51119d-b157-4465-a1be-41cb5e5ca063 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:21:34.510148 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:21:34.562152 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 756/1508] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:21:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:21:34.573477 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd87e12a-f49a-4624-a288-809a27c06365 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:21:34.574859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-bd87e12a-f49a-4624-a288-809a27c06365 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:21:34.662417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd87e12a-f49a-4624-a288-809a27c06365 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:21:34.666283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dde0c3-f1f4-4e31-9466-5e26ae87bc8b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:21:34.666894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dde0c3-f1f4-4e31-9466-5e26ae87bc8b service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:34.667173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dde0c3-f1f4-4e31-9466-5e26ae87bc8b service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:34.667450 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dde0c3-f1f4-4e31-9466-5e26ae87bc8b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:34.677746 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0dde0c3-f1f4-4e31-9466-5e26ae87bc8b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:21:34.679184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0dde0c3-f1f4-4e31-9466-5e26ae87bc8b service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:21:34.689039 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 753/1509] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:21:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:21:35.242278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef51119d-b157-4465-a1be-41cb5e5ca063 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:21:35.243832 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ef51119d-b157-4465-a1be-41cb5e5ca063 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:21:35.258458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef51119d-b157-4465-a1be-41cb5e5ca063 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:21:35.261444 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f89dc7-d841-4fec-889c-1976d2375248 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:21:35.266915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f89dc7-d841-4fec-889c-1976d2375248 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:21:35.266915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f89dc7-d841-4fec-889c-1976d2375248 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:21:35.266915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f89dc7-d841-4fec-889c-1976d2375248 admin admin] PATH_INFO: `/v3/projects/8e9618149b0b46808c35bf93ff4a0b19` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:21:35.266915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f89dc7-d841-4fec-889c-1976d2375248 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e9618149b0b46808c35bf93ff4a0b19)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:21:35.266915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: WARNING py.warnings [None req-03f89dc7-d841-4fec-889c-1976d2375248 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 21 01:21:35.266915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: warnings.warn(msg) Aug 21 01:21:35.266915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: Aug 21 01:21:35.270091 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f89dc7-d841-4fec-889c-1976d2375248 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:21:35.440001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-03f89dc7-d841-4fec-889c-1976d2375248 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e9618149b0b46808c35bf93ff4a0b19', 'request_id': 'req-03f89dc7-d841-4fec-889c-1976d2375248'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:21:35.446573 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-03f89dc7-d841-4fec-889c-1976d2375248 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e9618149b0b46808c35bf93ff4a0b19', 'request_id': 'req-03f89dc7-d841-4fec-889c-1976d2375248'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:21:35.460396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-03f89dc7-d841-4fec-889c-1976d2375248 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e9618149b0b46808c35bf93ff4a0b19', 'request_id': 'req-03f89dc7-d841-4fec-889c-1976d2375248'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:21:35.548085 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-03f89dc7-d841-4fec-889c-1976d2375248 admin admin] The token cache is being invalidate because project 8e9618149b0b46808c35bf93ff4a0b19 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:21:35.548345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-03f89dc7-d841-4fec-889c-1976d2375248 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:21:35.549045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-03f89dc7-d841-4fec-889c-1976d2375248 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:21:35.550548 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 757/1510] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:21:35 2022] DELETE /identity/v3/projects/8e9618149b0b46808c35bf93ff4a0b19 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:21:44.684619 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:DELETE=1,keystone:SELECT=37 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:21:45.544704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=47,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:23:02.139021 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d183a63a-7d06-4ceb-bce6-f782b1b579ff None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:02.139632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d183a63a-7d06-4ceb-bce6-f782b1b579ff None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:02.139632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d183a63a-7d06-4ceb-bce6-f782b1b579ff None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:02.140957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 754/1511] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 21 01:23:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:23:02.145933 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418929a-cddb-42b3-8a98-729c5573883e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:02.146474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418929a-cddb-42b3-8a98-729c5573883e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:02.146474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418929a-cddb-42b3-8a98-729c5573883e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:02.165692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-7418929a-cddb-42b3-8a98-729c5573883e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:23:02.172596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7418929a-cddb-42b3-8a98-729c5573883e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:23:02.228986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 758/1512] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 21 01:23:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:23:02.384686 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d183a63a-7d06-4ceb-bce6-f782b1b579ff None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:23:02.386406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-d183a63a-7d06-4ceb-bce6-f782b1b579ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:23:02.493107 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d183a63a-7d06-4ceb-bce6-f782b1b579ff None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:23:02.498703 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de232df-3487-43e8-9044-58d5bd037c44 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:23:02.499308 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de232df-3487-43e8-9044-58d5bd037c44 service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:02.499583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de232df-3487-43e8-9044-58d5bd037c44 service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:02.499854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de232df-3487-43e8-9044-58d5bd037c44 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:02.508310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de232df-3487-43e8-9044-58d5bd037c44 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:23:02.509901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de232df-3487-43e8-9044-58d5bd037c44 service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:23:02.519379 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 755/1513] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:23:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:23:03.359116 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ff0ca6-6870-4de0-9f70-ef1e299d5b4f None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:03.359586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ff0ca6-6870-4de0-9f70-ef1e299d5b4f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:03.359586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ff0ca6-6870-4de0-9f70-ef1e299d5b4f None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:03.360972 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 759/1514] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 21 01:23:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:23:03.365943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7768ea4a-388f-4551-bd73-d4bfaa7ad056 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:03.366414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7768ea4a-388f-4551-bd73-d4bfaa7ad056 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:03.366517 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7768ea4a-388f-4551-bd73-d4bfaa7ad056 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:03.386474 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-7768ea4a-388f-4551-bd73-d4bfaa7ad056 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:23:03.387683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-7768ea4a-388f-4551-bd73-d4bfaa7ad056 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:23:03.395536 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 756/1515] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 21 01:23:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:23:03.545928 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ff0ca6-6870-4de0-9f70-ef1e299d5b4f None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:23:03.563099 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ff0ca6-6870-4de0-9f70-ef1e299d5b4f None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:23:03.569572 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b242675c-3d45-4e3c-b127-7b9dce26ea3f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:23:03.570196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b242675c-3d45-4e3c-b127-7b9dce26ea3f service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:03.571057 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b242675c-3d45-4e3c-b127-7b9dce26ea3f service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:03.571057 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b242675c-3d45-4e3c-b127-7b9dce26ea3f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:03.579539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b242675c-3d45-4e3c-b127-7b9dce26ea3f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:23:03.581325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b242675c-3d45-4e3c-b127-7b9dce26ea3f service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:23:03.591583 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 760/1516] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:23:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:23:04.212997 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daae5a74-917c-479d-a1d7-1999a104847a None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:04.213276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daae5a74-917c-479d-a1d7-1999a104847a None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:04.213541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daae5a74-917c-479d-a1d7-1999a104847a None None] PATH_INFO: `/v3` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:04.214853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 757/1517] 10.222.0.67 () {54 vars in 925 bytes} [Sun Aug 21 01:23:04 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:23:04.223450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87679b82-bee6-4447-8c5e-2ab9315c8642 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:04.223732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87679b82-bee6-4447-8c5e-2ab9315c8642 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:04.224593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87679b82-bee6-4447-8c5e-2ab9315c8642 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:04.279059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-87679b82-bee6-4447-8c5e-2ab9315c8642 None None] MFA Rules not processed for user `972b0f1dbfef4d85a6dfc4c5fd839bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:23:04.280317 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-87679b82-bee6-4447-8c5e-2ab9315c8642 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:23:04.355707 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 761/1518] 10.222.0.67 () {58 vars in 1015 bytes} [Sun Aug 21 01:23:04 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:23:13.380747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=49 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:23:14.119320 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7393147b-95fc-4cc1-9658-10eb1e0e3e07 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:14.119534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7393147b-95fc-4cc1-9658-10eb1e0e3e07 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:14.119777 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7393147b-95fc-4cc1-9658-10eb1e0e3e07 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:14.121158 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 758/1519] 10.222.0.47 () {54 vars in 961 bytes} [Sun Aug 21 01:23:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:23:14.126264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520ff751-3b2e-47a7-a90b-d3d9e51d1eea None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:14.126752 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520ff751-3b2e-47a7-a90b-d3d9e51d1eea None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:14.126846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520ff751-3b2e-47a7-a90b-d3d9e51d1eea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:14.147820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-520ff751-3b2e-47a7-a90b-d3d9e51d1eea None None] MFA Rules not processed for user `972b0f1dbfef4d85a6dfc4c5fd839bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:23:14.149082 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-520ff751-3b2e-47a7-a90b-d3d9e51d1eea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:23:14.157732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 762/1520] 10.222.0.47 () {58 vars in 1058 bytes} [Sun Aug 21 01:23:14 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:23:14.166524 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7393147b-95fc-4cc1-9658-10eb1e0e3e07 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:23:14.181685 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7393147b-95fc-4cc1-9658-10eb1e0e3e07 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:23:14.186616 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b4d309-5849-49cb-862b-b313c45e51eb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '972b0f1dbfef4d85a6dfc4c5fd839bde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:23:14.187218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b4d309-5849-49cb-862b-b313c45e51eb service placement] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:14.187493 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b4d309-5849-49cb-862b-b313c45e51eb service placement] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:14.187760 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b4d309-5849-49cb-862b-b313c45e51eb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:14.197493 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b4d309-5849-49cb-862b-b313c45e51eb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:23:14.199081 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b4d309-5849-49cb-862b-b313c45e51eb service placement] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:23:14.210879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 759/1521] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:23:14 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:23:14.814505 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31cb560-1a29-49b3-a5a9-e1ba50619752 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:14.814922 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31cb560-1a29-49b3-a5a9-e1ba50619752 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:14.815293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31cb560-1a29-49b3-a5a9-e1ba50619752 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:14.816829 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 763/1522] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 21 01:23:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:23:14.822020 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4266253d-2030-498c-87d7-a7a3e47a5ddb None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:14.822477 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4266253d-2030-498c-87d7-a7a3e47a5ddb None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:14.822590 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4266253d-2030-498c-87d7-a7a3e47a5ddb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:14.842324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-4266253d-2030-498c-87d7-a7a3e47a5ddb None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:23:14.843507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-4266253d-2030-498c-87d7-a7a3e47a5ddb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:23:14.851293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 760/1523] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 21 01:23:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:23:15.011359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31cb560-1a29-49b3-a5a9-e1ba50619752 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:23:15.024879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31cb560-1a29-49b3-a5a9-e1ba50619752 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:23:15.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14df424-c6ac-4843-9648-565ceabc4d12 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:23:15.031124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14df424-c6ac-4843-9648-565ceabc4d12 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:15.031404 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14df424-c6ac-4843-9648-565ceabc4d12 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:15.031699 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14df424-c6ac-4843-9648-565ceabc4d12 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:15.040258 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14df424-c6ac-4843-9648-565ceabc4d12 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:23:15.042389 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14df424-c6ac-4843-9648-565ceabc4d12 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:23:15.052447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 764/1524] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:23:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:23:15.880778 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d3b795-7702-439e-85f8-8480531dd679 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:15.881358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d3b795-7702-439e-85f8-8480531dd679 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:15.881358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d3b795-7702-439e-85f8-8480531dd679 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:15.882697 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 761/1525] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 21 01:23:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:23:15.887743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb4d435-45fe-4812-bb14-dc3b36fec6bc None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:15.888220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb4d435-45fe-4812-bb14-dc3b36fec6bc None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:15.888320 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb4d435-45fe-4812-bb14-dc3b36fec6bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:15.914504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-abb4d435-45fe-4812-bb14-dc3b36fec6bc None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:23:15.915693 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-abb4d435-45fe-4812-bb14-dc3b36fec6bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:23:15.924260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 765/1526] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 21 01:23:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:23:16.076451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d3b795-7702-439e-85f8-8480531dd679 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:23:16.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d3b795-7702-439e-85f8-8480531dd679 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:23:16.105258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1d59ee-e49a-416e-9b8d-d5ac68a0afa5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:23:16.105872 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1d59ee-e49a-416e-9b8d-d5ac68a0afa5 service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:16.106150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1d59ee-e49a-416e-9b8d-d5ac68a0afa5 service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:16.106417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1d59ee-e49a-416e-9b8d-d5ac68a0afa5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:16.117399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1d59ee-e49a-416e-9b8d-d5ac68a0afa5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:23:16.119126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1d59ee-e49a-416e-9b8d-d5ac68a0afa5 service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:23:16.129803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 762/1527] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:23:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:23:19.981230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231f501c-43dc-4d17-b564-c42157f1c5e3 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:19.981800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231f501c-43dc-4d17-b564-c42157f1c5e3 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:19.981800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231f501c-43dc-4d17-b564-c42157f1c5e3 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:19.983264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 766/1528] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 21 01:23:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:23:19.988883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a4b2fd-f945-4c1a-ba20-03252b1af545 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:19.989333 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a4b2fd-f945-4c1a-ba20-03252b1af545 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:19.989431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a4b2fd-f945-4c1a-ba20-03252b1af545 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:20.009772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-54a4b2fd-f945-4c1a-ba20-03252b1af545 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:23:20.011002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-54a4b2fd-f945-4c1a-ba20-03252b1af545 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:23:20.019042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 763/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 21 01:23:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:23:20.325221 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231f501c-43dc-4d17-b564-c42157f1c5e3 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:23:20.326919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-231f501c-43dc-4d17-b564-c42157f1c5e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:23:20.356317 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:23:20.388230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231f501c-43dc-4d17-b564-c42157f1c5e3 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:23:20.392323 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279fb8c1-641e-4c09-822f-51816adbd523 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:23:20.392937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279fb8c1-641e-4c09-822f-51816adbd523 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:20.393240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279fb8c1-641e-4c09-822f-51816adbd523 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:20.393517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279fb8c1-641e-4c09-822f-51816adbd523 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:20.402954 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279fb8c1-641e-4c09-822f-51816adbd523 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:23:20.404419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279fb8c1-641e-4c09-822f-51816adbd523 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:23:20.414544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 767/1530] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:23:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:23:22.356626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1625d492-3630-4aea-953a-14e66397f90a None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:22.357173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1625d492-3630-4aea-953a-14e66397f90a None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:22.357173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1625d492-3630-4aea-953a-14e66397f90a None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:22.358508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 764/1531] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 21 01:23:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:23:22.364586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68868130-2574-4da2-be13-fd2e5b82d676 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:22.365081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68868130-2574-4da2-be13-fd2e5b82d676 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:22.365188 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68868130-2574-4da2-be13-fd2e5b82d676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:22.385845 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-68868130-2574-4da2-be13-fd2e5b82d676 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:23:22.387092 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-68868130-2574-4da2-be13-fd2e5b82d676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:23:22.395187 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 768/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 21 01:23:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:23:22.683912 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1625d492-3630-4aea-953a-14e66397f90a None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:23:22.685760 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1625d492-3630-4aea-953a-14e66397f90a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:23:22.699231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1625d492-3630-4aea-953a-14e66397f90a None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:23:22.703522 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf9dd4c-f4f1-413b-aca2-4840ede68037 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:23:22.704273 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf9dd4c-f4f1-413b-aca2-4840ede68037 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:22.704551 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf9dd4c-f4f1-413b-aca2-4840ede68037 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:22.704818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf9dd4c-f4f1-413b-aca2-4840ede68037 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:22.715439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf9dd4c-f4f1-413b-aca2-4840ede68037 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:23:22.716857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf9dd4c-f4f1-413b-aca2-4840ede68037 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:23:22.727564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 765/1533] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:23:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:23:26.616219 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f574b21-9dea-40d2-9367-5c626fafdf43 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:26.616772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f574b21-9dea-40d2-9367-5c626fafdf43 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:26.616772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f574b21-9dea-40d2-9367-5c626fafdf43 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:26.618130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 769/1534] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 21 01:23:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:23:26.623607 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff716e57-80a9-4ebc-b7b1-a18f8c5825b6 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:26.624034 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff716e57-80a9-4ebc-b7b1-a18f8c5825b6 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:26.624211 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff716e57-80a9-4ebc-b7b1-a18f8c5825b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:26.643864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-ff716e57-80a9-4ebc-b7b1-a18f8c5825b6 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:23:26.645096 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ff716e57-80a9-4ebc-b7b1-a18f8c5825b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:23:26.653195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 766/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 21 01:23:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:23:27.216935 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f574b21-9dea-40d2-9367-5c626fafdf43 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:23:27.218820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-1f574b21-9dea-40d2-9367-5c626fafdf43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:23:27.277026 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f574b21-9dea-40d2-9367-5c626fafdf43 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:23:27.280624 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febd08b8-aa67-4002-a7f0-ef5139466a73 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:23:27.281233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febd08b8-aa67-4002-a7f0-ef5139466a73 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:27.281512 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febd08b8-aa67-4002-a7f0-ef5139466a73 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:27.281778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febd08b8-aa67-4002-a7f0-ef5139466a73 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:27.291262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febd08b8-aa67-4002-a7f0-ef5139466a73 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:23:27.292722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febd08b8-aa67-4002-a7f0-ef5139466a73 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:23:27.303058 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 770/1536] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:23:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:23:29.234342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc43057-904e-4427-bce4-cb86b7f80862 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:29.234907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc43057-904e-4427-bce4-cb86b7f80862 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:29.234907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc43057-904e-4427-bce4-cb86b7f80862 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:29.236381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 767/1537] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 21 01:23:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:23:29.242364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a057e215-3e3d-40f8-9d11-539ff7f6368e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:29.242829 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a057e215-3e3d-40f8-9d11-539ff7f6368e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:29.242938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a057e215-3e3d-40f8-9d11-539ff7f6368e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:29.262890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-a057e215-3e3d-40f8-9d11-539ff7f6368e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:23:29.264122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a057e215-3e3d-40f8-9d11-539ff7f6368e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:23:29.272753 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 771/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 21 01:23:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:23:29.835651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc43057-904e-4427-bce4-cb86b7f80862 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:23:29.837395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ebc43057-904e-4427-bce4-cb86b7f80862 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:23:29.851263 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc43057-904e-4427-bce4-cb86b7f80862 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:23:29.855067 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3457c1c8-011d-445a-8cc8-23224f56a854 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:23:29.855663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457c1c8-011d-445a-8cc8-23224f56a854 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:23:29.855940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457c1c8-011d-445a-8cc8-23224f56a854 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:23:29.856271 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457c1c8-011d-445a-8cc8-23224f56a854 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:23:29.865462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3457c1c8-011d-445a-8cc8-23224f56a854 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:23:29.866889 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3457c1c8-011d-445a-8cc8-23224f56a854 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:23:29.876912 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 768/1539] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:23:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:23:39.258437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=47 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:23:39.872462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=33 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:24:55.037683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abdb1a8-b8c8-4527-b142-76d799b9a008 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.067940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abdb1a8-b8c8-4527-b142-76d799b9a008 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.067940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abdb1a8-b8c8-4527-b142-76d799b9a008 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.068420 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9f1e8-d440-4afb-a228-cfb43f3c2024 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.068420 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9f1e8-d440-4afb-a228-cfb43f3c2024 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.068420 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9f1e8-d440-4afb-a228-cfb43f3c2024 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.071435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-9de9f1e8-d440-4afb-a228-cfb43f3c2024 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:55.071435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9de9f1e8-d440-4afb-a228-cfb43f3c2024 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:55.080544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-2abdb1a8-b8c8-4527-b142-76d799b9a008 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:55.082552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-2abdb1a8-b8c8-4527-b142-76d799b9a008 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:55.085814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 769/1540] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:55.094791 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e234e446-b26e-4504-858c-da6b4f84c11f None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.095162 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e234e446-b26e-4504-858c-da6b4f84c11f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.095371 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e234e446-b26e-4504-858c-da6b4f84c11f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.098572 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 772/1541] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:24:55.108568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf40c53-9a4a-4584-ad35-0d7dd2075728 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.109215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf40c53-9a4a-4584-ad35-0d7dd2075728 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.109704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf40c53-9a4a-4584-ad35-0d7dd2075728 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.123986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-e234e446-b26e-4504-858c-da6b4f84c11f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:55.125234 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e234e446-b26e-4504-858c-da6b4f84c11f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:55.140612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 770/1542] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:24:55.143980 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e234e446-b26e-4504-858c-da6b4f84c11f None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:55.154012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-8cf40c53-9a4a-4584-ad35-0d7dd2075728 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:55.155937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8cf40c53-9a4a-4584-ad35-0d7dd2075728 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:55.166857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e234e446-b26e-4504-858c-da6b4f84c11f None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:55.177031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5a86ed-ca0e-43ee-8e0e-4ed74758556a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:55.178067 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5a86ed-ca0e-43ee-8e0e-4ed74758556a admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.178509 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5a86ed-ca0e-43ee-8e0e-4ed74758556a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.178943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5a86ed-ca0e-43ee-8e0e-4ed74758556a admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.179765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5a86ed-ca0e-43ee-8e0e-4ed74758556a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:24:55.180342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5a86ed-ca0e-43ee-8e0e-4ed74758556a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:55.182111 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: WARNING py.warnings [None req-4b5a86ed-ca0e-43ee-8e0e-4ed74758556a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 21 01:24:55.182111 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: warnings.warn(msg) Aug 21 01:24:55.182111 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: Aug 21 01:24:55.183362 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5a86ed-ca0e-43ee-8e0e-4ed74758556a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:55.183496 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 773/1543] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:24:55.187921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a85401-1685-490e-a6f7-7831e429b5db None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.187921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a85401-1685-490e-a6f7-7831e429b5db None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.187921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a85401-1685-490e-a6f7-7831e429b5db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.208534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 771/1544] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:24:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:24:55.209964 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5a86ed-ca0e-43ee-8e0e-4ed74758556a admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:55.225414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5a86ed-ca0e-43ee-8e0e-4ed74758556a admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:55.237428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25bf64ed-f034-439b-bc9c-0e255ecfa9bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:55.238050 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-00a85401-1685-490e-a6f7-7831e429b5db None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:55.238694 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bf64ed-f034-439b-bc9c-0e255ecfa9bf admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.239188 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bf64ed-f034-439b-bc9c-0e255ecfa9bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.239314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-00a85401-1685-490e-a6f7-7831e429b5db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:55.239796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bf64ed-f034-439b-bc9c-0e255ecfa9bf admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.240787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25bf64ed-f034-439b-bc9c-0e255ecfa9bf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:24:55.241327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25bf64ed-f034-439b-bc9c-0e255ecfa9bf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:55.243304 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25bf64ed-f034-439b-bc9c-0e255ecfa9bf admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:55.261956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 774/1545] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:55.264067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e007a67-808a-41ce-bd91-0ffcda6476ad None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.264312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e007a67-808a-41ce-bd91-0ffcda6476ad None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.264546 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e007a67-808a-41ce-bd91-0ffcda6476ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.265203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 772/1546] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:24:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:24:55.272127 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25bf64ed-f034-439b-bc9c-0e255ecfa9bf admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:55.286161 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25bf64ed-f034-439b-bc9c-0e255ecfa9bf admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:55.296826 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72edaf6-0e87-4d0d-b21c-31b69d1dd212 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:55.297795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72edaf6-0e87-4d0d-b21c-31b69d1dd212 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.298305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72edaf6-0e87-4d0d-b21c-31b69d1dd212 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.298792 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72edaf6-0e87-4d0d-b21c-31b69d1dd212 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.299656 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72edaf6-0e87-4d0d-b21c-31b69d1dd212 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:55.301928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72edaf6-0e87-4d0d-b21c-31b69d1dd212 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:55.314005 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-7e007a67-808a-41ce-bd91-0ffcda6476ad None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:55.315303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7e007a67-808a-41ce-bd91-0ffcda6476ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:55.326831 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 775/1547] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:55.327995 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6d0635-5138-48bf-9358-6a408ebfcb54 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.328328 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6d0635-5138-48bf-9358-6a408ebfcb54 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.332332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6d0635-5138-48bf-9358-6a408ebfcb54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.351855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-cf6d0635-5138-48bf-9358-6a408ebfcb54 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:55.354227 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-cf6d0635-5138-48bf-9358-6a408ebfcb54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:55.363954 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 776/1548] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:55.367955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6d0635-5138-48bf-9358-6a408ebfcb54 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:55.376757 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6d0635-5138-48bf-9358-6a408ebfcb54 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:55.382098 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0067ded1-011b-4f2f-8c8d-9882a2740000 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:55.382730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0067ded1-011b-4f2f-8c8d-9882a2740000 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.383008 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0067ded1-011b-4f2f-8c8d-9882a2740000 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.383280 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0067ded1-011b-4f2f-8c8d-9882a2740000 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.384229 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0067ded1-011b-4f2f-8c8d-9882a2740000 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:55.385355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: WARNING py.warnings [None req-0067ded1-011b-4f2f-8c8d-9882a2740000 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 21 01:24:55.385355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: warnings.warn(msg) Aug 21 01:24:55.385355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: Aug 21 01:24:55.386614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0067ded1-011b-4f2f-8c8d-9882a2740000 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:55.408115 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 773/1549] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/projects => generated 383 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:55.414513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea355eed-f47e-4dfb-aa15-7ebc0cce01c5 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.414513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea355eed-f47e-4dfb-aa15-7ebc0cce01c5 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.414513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea355eed-f47e-4dfb-aa15-7ebc0cce01c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.433849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-ea355eed-f47e-4dfb-aa15-7ebc0cce01c5 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:55.435057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ea355eed-f47e-4dfb-aa15-7ebc0cce01c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:55.493353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 777/1550] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/projects => generated 393 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:55.494928 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5e7e30-de0d-45af-b663-a46164b754e3 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.495207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5e7e30-de0d-45af-b663-a46164b754e3 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.495474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5e7e30-de0d-45af-b663-a46164b754e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.500974 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 774/1551] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:55.502540 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583d26fa-8c23-43b7-823f-ae0dea353de1 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.502918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583d26fa-8c23-43b7-823f-ae0dea353de1 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.503289 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583d26fa-8c23-43b7-823f-ae0dea353de1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.526724 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-1d5e7e30-de0d-45af-b663-a46164b754e3 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:55.528849 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-1d5e7e30-de0d-45af-b663-a46164b754e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:55.529248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-583d26fa-8c23-43b7-823f-ae0dea353de1 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:55.530452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-583d26fa-8c23-43b7-823f-ae0dea353de1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:55.596333 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 775/1552] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:55.596536 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 778/1553] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:55.596656 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583d26fa-8c23-43b7-823f-ae0dea353de1 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:55.596763 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5e7e30-de0d-45af-b663-a46164b754e3 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:55.605700 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5e7e30-de0d-45af-b663-a46164b754e3 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:55.607349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583d26fa-8c23-43b7-823f-ae0dea353de1 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:55.610662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46d72cd-6e2f-46a6-9e3e-6f2b399ec671 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:55.611264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46d72cd-6e2f-46a6-9e3e-6f2b399ec671 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.611681 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46d72cd-6e2f-46a6-9e3e-6f2b399ec671 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.611955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46d72cd-6e2f-46a6-9e3e-6f2b399ec671 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.612176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1bbcd7-c6fa-479a-8111-27b8cc436d99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:55.612639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46d72cd-6e2f-46a6-9e3e-6f2b399ec671 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:55.612757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1bbcd7-c6fa-479a-8111-27b8cc436d99 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.612907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1bbcd7-c6fa-479a-8111-27b8cc436d99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.613173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1bbcd7-c6fa-479a-8111-27b8cc436d99 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.613815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1bbcd7-c6fa-479a-8111-27b8cc436d99 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:24:55.614106 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1bbcd7-c6fa-479a-8111-27b8cc436d99 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:55.614574 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46d72cd-6e2f-46a6-9e3e-6f2b399ec671 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:55.615675 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1bbcd7-c6fa-479a-8111-27b8cc436d99 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:55.628407 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 776/1554] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:24:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:24:55.629711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a5da66-02d6-4d53-adc0-ab095d59760c None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.630014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a5da66-02d6-4d53-adc0-ab095d59760c None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.630272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a5da66-02d6-4d53-adc0-ab095d59760c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.653822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-68a5da66-02d6-4d53-adc0-ab095d59760c None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:55.655058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-68a5da66-02d6-4d53-adc0-ab095d59760c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:55.663070 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 777/1555] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:55.667158 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a5da66-02d6-4d53-adc0-ab095d59760c None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:55.674460 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a5da66-02d6-4d53-adc0-ab095d59760c None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:55.679106 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f38846-27a8-48ef-9916-e02b0f9a3328 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:55.679713 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f38846-27a8-48ef-9916-e02b0f9a3328 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.679987 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f38846-27a8-48ef-9916-e02b0f9a3328 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.680292 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f38846-27a8-48ef-9916-e02b0f9a3328 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.680915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f38846-27a8-48ef-9916-e02b0f9a3328 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:55.682010 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: WARNING py.warnings [None req-19f38846-27a8-48ef-9916-e02b0f9a3328 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 21 01:24:55.682010 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: warnings.warn(msg) Aug 21 01:24:55.682010 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: Aug 21 01:24:55.683218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f38846-27a8-48ef-9916-e02b0f9a3328 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:55.701574 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 779/1556] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/users => generated 292 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:55.704742 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94656ffb-e7ea-4cfa-af63-48098f65ed19 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.704742 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94656ffb-e7ea-4cfa-af63-48098f65ed19 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.704742 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94656ffb-e7ea-4cfa-af63-48098f65ed19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.725255 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-94656ffb-e7ea-4cfa-af63-48098f65ed19 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:55.726561 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-94656ffb-e7ea-4cfa-af63-48098f65ed19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:55.739628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 780/1557] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:55.745024 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47137f73-7b9f-453a-9fd6-2efa0d56514c None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.745024 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47137f73-7b9f-453a-9fd6-2efa0d56514c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.745024 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47137f73-7b9f-453a-9fd6-2efa0d56514c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.763296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-47137f73-7b9f-453a-9fd6-2efa0d56514c None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:55.776179 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-47137f73-7b9f-453a-9fd6-2efa0d56514c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:55.776354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 778/1558] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/users => generated 297 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:55.776354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f38846-27a8-48ef-9916-e02b0f9a3328 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:55.776354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:24:55.776608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 781/1559] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:55.776608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846149bd-7af4-4803-b5cb-e799f3f1e0aa None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.776608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846149bd-7af4-4803-b5cb-e799f3f1e0aa None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.776608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846149bd-7af4-4803-b5cb-e799f3f1e0aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.780820 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f38846-27a8-48ef-9916-e02b0f9a3328 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:55.785898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74211d8-d0bd-4906-8c4b-4678d72ab7a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:55.786498 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74211d8-d0bd-4906-8c4b-4678d72ab7a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.786769 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74211d8-d0bd-4906-8c4b-4678d72ab7a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.787039 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74211d8-d0bd-4906-8c4b-4678d72ab7a6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.787676 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74211d8-d0bd-4906-8c4b-4678d72ab7a6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:24:55.787962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74211d8-d0bd-4906-8c4b-4678d72ab7a6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:55.789581 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74211d8-d0bd-4906-8c4b-4678d72ab7a6 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:55.795656 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-846149bd-7af4-4803-b5cb-e799f3f1e0aa None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:55.796910 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-846149bd-7af4-4803-b5cb-e799f3f1e0aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:55.802266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 779/1560] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:24:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:24:55.803166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74211d8-d0bd-4906-8c4b-4678d72ab7a6 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:55.808859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 782/1561] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:55.808859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51dc096-6689-40e0-9cbc-d64df13f2605 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.808859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51dc096-6689-40e0-9cbc-d64df13f2605 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.808859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51dc096-6689-40e0-9cbc-d64df13f2605 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.814740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74211d8-d0bd-4906-8c4b-4678d72ab7a6 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:55.819806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481d0aa3-6274-4303-937a-b9fdc319b194 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:55.820454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481d0aa3-6274-4303-937a-b9fdc319b194 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.820783 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481d0aa3-6274-4303-937a-b9fdc319b194 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.821237 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481d0aa3-6274-4303-937a-b9fdc319b194 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.821829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481d0aa3-6274-4303-937a-b9fdc319b194 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:55.824263 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481d0aa3-6274-4303-937a-b9fdc319b194 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:55.833257 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-d51dc096-6689-40e0-9cbc-d64df13f2605 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:55.834549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d51dc096-6689-40e0-9cbc-d64df13f2605 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:55.846230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 783/1562] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:55.847129 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51dc096-6689-40e0-9cbc-d64df13f2605 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:55.857212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51dc096-6689-40e0-9cbc-d64df13f2605 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:55.863914 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198e2a48-4a02-4fa8-a3ac-29d69943bbb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:55.864545 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198e2a48-4a02-4fa8-a3ac-29d69943bbb4 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.864822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198e2a48-4a02-4fa8-a3ac-29d69943bbb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.865092 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198e2a48-4a02-4fa8-a3ac-29d69943bbb4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.865748 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198e2a48-4a02-4fa8-a3ac-29d69943bbb4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:55.867343 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198e2a48-4a02-4fa8-a3ac-29d69943bbb4 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:55.877957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 784/1563] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:55 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:55.878849 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198e2a48-4a02-4fa8-a3ac-29d69943bbb4 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:55.890938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198e2a48-4a02-4fa8-a3ac-29d69943bbb4 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:55.895555 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943618c2-2554-42f2-b3b6-9fed9e10697e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:55.896165 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943618c2-2554-42f2-b3b6-9fed9e10697e admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.896440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943618c2-2554-42f2-b3b6-9fed9e10697e admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.896706 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943618c2-2554-42f2-b3b6-9fed9e10697e admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.897329 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943618c2-2554-42f2-b3b6-9fed9e10697e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:24:55.897622 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943618c2-2554-42f2-b3b6-9fed9e10697e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:55.899171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943618c2-2554-42f2-b3b6-9fed9e10697e admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:55.912091 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 785/1564] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:24:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:24:55.912890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943618c2-2554-42f2-b3b6-9fed9e10697e admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:55.922605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943618c2-2554-42f2-b3b6-9fed9e10697e admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:55.927300 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c9f616-24bc-4d8f-bab6-616a9bf38873 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:55.927899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c9f616-24bc-4d8f-bab6-616a9bf38873 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.928198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c9f616-24bc-4d8f-bab6-616a9bf38873 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.928467 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c9f616-24bc-4d8f-bab6-616a9bf38873 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.929083 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c9f616-24bc-4d8f-bab6-616a9bf38873 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:55.930674 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c9f616-24bc-4d8f-bab6-616a9bf38873 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:55.940646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 786/1565] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:55 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:55.941779 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d76637-4432-4f79-ba11-33a703f6de5b None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.942057 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d76637-4432-4f79-ba11-33a703f6de5b None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.942326 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d76637-4432-4f79-ba11-33a703f6de5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.963817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-66d76637-4432-4f79-ba11-33a703f6de5b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:55.965067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-66d76637-4432-4f79-ba11-33a703f6de5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:55.981480 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 787/1566] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:55.982311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d76637-4432-4f79-ba11-33a703f6de5b None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:55.991860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d76637-4432-4f79-ba11-33a703f6de5b None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:55.997655 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253649d0-be17-496a-953e-76fbab72b9f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:55.998260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253649d0-be17-496a-953e-76fbab72b9f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:55.998532 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253649d0-be17-496a-953e-76fbab72b9f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:55.998800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253649d0-be17-496a-953e-76fbab72b9f1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:55.999431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253649d0-be17-496a-953e-76fbab72b9f1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:24:55.999732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253649d0-be17-496a-953e-76fbab72b9f1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:56.001557 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253649d0-be17-496a-953e-76fbab72b9f1 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:56.014337 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 788/1567] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:24:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:24:56.015279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253649d0-be17-496a-953e-76fbab72b9f1 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:56.024910 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253649d0-be17-496a-953e-76fbab72b9f1 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:56.030075 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f8c109-2e5f-4e46-81e4-3beac22909d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:56.030646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f8c109-2e5f-4e46-81e4-3beac22909d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:56.030916 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f8c109-2e5f-4e46-81e4-3beac22909d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:56.031177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f8c109-2e5f-4e46-81e4-3beac22909d1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:56.031801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f8c109-2e5f-4e46-81e4-3beac22909d1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:56.033714 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f8c109-2e5f-4e46-81e4-3beac22909d1 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:56.038156 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:24:56.049190 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 780/1568] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:24:55 2022] POST /identity/v3/projects => generated 375 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:56.051982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481d0aa3-6274-4303-937a-b9fdc319b194 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:56.127972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481d0aa3-6274-4303-937a-b9fdc319b194 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:56.132962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d697cef-9c14-40b8-a44a-68c807e7398f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:56.133596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d697cef-9c14-40b8-a44a-68c807e7398f admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:56.133872 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d697cef-9c14-40b8-a44a-68c807e7398f admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:56.134142 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d697cef-9c14-40b8-a44a-68c807e7398f admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:56.135430 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d697cef-9c14-40b8-a44a-68c807e7398f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:24:56.135733 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d697cef-9c14-40b8-a44a-68c807e7398f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:56.137484 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d697cef-9c14-40b8-a44a-68c807e7398f admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:56.149425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 781/1569] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:24:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:24:56.150324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d697cef-9c14-40b8-a44a-68c807e7398f admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:56.163540 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d697cef-9c14-40b8-a44a-68c807e7398f admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:56.168691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a96a1fb-c57c-4d30-bacf-4b1dc3512ca0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:56.171473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a96a1fb-c57c-4d30-bacf-4b1dc3512ca0 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:56.171473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a96a1fb-c57c-4d30-bacf-4b1dc3512ca0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:56.171473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a96a1fb-c57c-4d30-bacf-4b1dc3512ca0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:56.171473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a96a1fb-c57c-4d30-bacf-4b1dc3512ca0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:56.172206 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a96a1fb-c57c-4d30-bacf-4b1dc3512ca0 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:56.181307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 789/1570] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:24:56 2022] POST /identity/v3/projects => generated 375 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:56.181307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f8c109-2e5f-4e46-81e4-3beac22909d1 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:56.189143 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 782/1571] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:56 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:56.189143 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a96a1fb-c57c-4d30-bacf-4b1dc3512ca0 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:56.200050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a96a1fb-c57c-4d30-bacf-4b1dc3512ca0 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:56.259329 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f8c109-2e5f-4e46-81e4-3beac22909d1 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:56.265274 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b309d42-2a66-4e8a-af2d-672e5aa05184 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:56.265859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b309d42-2a66-4e8a-af2d-672e5aa05184 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:56.266137 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b309d42-2a66-4e8a-af2d-672e5aa05184 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:56.266407 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b309d42-2a66-4e8a-af2d-672e5aa05184 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:56.267041 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b309d42-2a66-4e8a-af2d-672e5aa05184 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:56.267227 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c738df19-6fc4-4cb5-ac2e-79be37ceaa6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:56.267918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c738df19-6fc4-4cb5-ac2e-79be37ceaa6c admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:56.268463 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c738df19-6fc4-4cb5-ac2e-79be37ceaa6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:56.268808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c738df19-6fc4-4cb5-ac2e-79be37ceaa6c admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:56.269560 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c738df19-6fc4-4cb5-ac2e-79be37ceaa6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:56.270197 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b309d42-2a66-4e8a-af2d-672e5aa05184 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:56.273794 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c738df19-6fc4-4cb5-ac2e-79be37ceaa6c admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:56.287043 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 783/1572] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:56 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:56.287043 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c738df19-6fc4-4cb5-ac2e-79be37ceaa6c admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:56.295683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c738df19-6fc4-4cb5-ac2e-79be37ceaa6c admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:56.301020 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f29398-dee3-413d-acc6-57728f3baa9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:56.301614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f29398-dee3-413d-acc6-57728f3baa9c admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:56.302046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f29398-dee3-413d-acc6-57728f3baa9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:56.302212 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f29398-dee3-413d-acc6-57728f3baa9c admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:56.302819 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f29398-dee3-413d-acc6-57728f3baa9c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:24:56.303118 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f29398-dee3-413d-acc6-57728f3baa9c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:56.304766 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f29398-dee3-413d-acc6-57728f3baa9c admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:56.320157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 784/1573] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:24:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:24:56.320974 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f29398-dee3-413d-acc6-57728f3baa9c admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:56.333832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f29398-dee3-413d-acc6-57728f3baa9c admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:56.344092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78ae087-cece-4c66-99fd-019d088b116c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:56.344092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78ae087-cece-4c66-99fd-019d088b116c admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:56.344092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78ae087-cece-4c66-99fd-019d088b116c admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:56.344092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78ae087-cece-4c66-99fd-019d088b116c admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:56.345176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78ae087-cece-4c66-99fd-019d088b116c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:56.345176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78ae087-cece-4c66-99fd-019d088b116c admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:56.411420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 790/1574] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:24:56 2022] POST /identity/v3/projects => generated 377 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:56.412680 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b309d42-2a66-4e8a-af2d-672e5aa05184 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:56.481314 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 785/1575] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:24:56 2022] POST /identity/v3/projects => generated 377 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:56.485723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78ae087-cece-4c66-99fd-019d088b116c admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:56.502162 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:24:56.507811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b309d42-2a66-4e8a-af2d-672e5aa05184 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:56.575888 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78ae087-cece-4c66-99fd-019d088b116c admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:56.583360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37062e08-6e6b-4521-a2e4-3df31b518bfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:56.584074 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37062e08-6e6b-4521-a2e4-3df31b518bfb admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:56.584685 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37062e08-6e6b-4521-a2e4-3df31b518bfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:56.585039 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37062e08-6e6b-4521-a2e4-3df31b518bfb admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:56.586101 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37062e08-6e6b-4521-a2e4-3df31b518bfb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:56.588579 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37062e08-6e6b-4521-a2e4-3df31b518bfb admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:56.597168 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6029e9-1bd3-4e81-afd2-c4bb5d98a36b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:56.598335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6029e9-1bd3-4e81-afd2-c4bb5d98a36b admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:56.598711 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6029e9-1bd3-4e81-afd2-c4bb5d98a36b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:56.599210 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6029e9-1bd3-4e81-afd2-c4bb5d98a36b admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:56.600616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6029e9-1bd3-4e81-afd2-c4bb5d98a36b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:56.609117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6029e9-1bd3-4e81-afd2-c4bb5d98a36b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:56.672259 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 786/1576] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:24:56 2022] POST /identity/v3/projects => generated 389 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:56.673130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37062e08-6e6b-4521-a2e4-3df31b518bfb admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:56.698816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 791/1577] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:24:56 2022] POST /identity/v3/users => generated 288 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:56.699631 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6029e9-1bd3-4e81-afd2-c4bb5d98a36b admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:56.711308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6029e9-1bd3-4e81-afd2-c4bb5d98a36b admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:56.716595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:24:56.743414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37062e08-6e6b-4521-a2e4-3df31b518bfb admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:56.750062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691639f6-c1bd-4280-a05b-c7a51ec7e0d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:56.756840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691639f6-c1bd-4280-a05b-c7a51ec7e0d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:56.756840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691639f6-c1bd-4280-a05b-c7a51ec7e0d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:56.756840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691639f6-c1bd-4280-a05b-c7a51ec7e0d1 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:56.756840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691639f6-c1bd-4280-a05b-c7a51ec7e0d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:56.756840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691639f6-c1bd-4280-a05b-c7a51ec7e0d1 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:56.783843 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b200286-19fe-4ba0-bb8a-a41fb34ae8be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:56.784791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b200286-19fe-4ba0-bb8a-a41fb34ae8be admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:56.785533 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b200286-19fe-4ba0-bb8a-a41fb34ae8be admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:56.785948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b200286-19fe-4ba0-bb8a-a41fb34ae8be admin admin] PATH_INFO: `/v3/projects/1f324851bf32439880d91a81160858ab/users/dc3e6a6484c74d2cab56e6696404817d/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:56.803446 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b200286-19fe-4ba0-bb8a-a41fb34ae8be admin admin] RBAC: Authorizing `identity:create_grant(project_id=1f324851bf32439880d91a81160858ab, user_id=dc3e6a6484c74d2cab56e6696404817d, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:56.804553 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: WARNING py.warnings [None req-5b200286-19fe-4ba0-bb8a-a41fb34ae8be admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 21 01:24:56.804553 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: warnings.warn(msg) Aug 21 01:24:56.804553 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: Aug 21 01:24:56.807321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b200286-19fe-4ba0-bb8a-a41fb34ae8be admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:56.864739 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 787/1578] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:24:56 2022] POST /identity/v3/users => generated 288 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:56.865647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691639f6-c1bd-4280-a05b-c7a51ec7e0d1 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:56.875575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691639f6-c1bd-4280-a05b-c7a51ec7e0d1 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:56.880309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a84842-92b3-4e6d-9935-b955f3dd0f47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:56.880950 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a84842-92b3-4e6d-9935-b955f3dd0f47 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:56.881227 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a84842-92b3-4e6d-9935-b955f3dd0f47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:56.881501 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a84842-92b3-4e6d-9935-b955f3dd0f47 admin admin] PATH_INFO: `/v3/projects/9774c354b8b048768f2071acfb48a4bf/users/d84b6d05decb47c7b55ac782d5839c1a/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:56.897491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 792/1579] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:24:56 2022] PUT /identity/v3/projects/1f324851bf32439880d91a81160858ab/users/dc3e6a6484c74d2cab56e6696404817d/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:24:56.898316 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b200286-19fe-4ba0-bb8a-a41fb34ae8be admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:56.901525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a84842-92b3-4e6d-9935-b955f3dd0f47 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9774c354b8b048768f2071acfb48a4bf, user_id=d84b6d05decb47c7b55ac782d5839c1a, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:56.905120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a84842-92b3-4e6d-9935-b955f3dd0f47 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:56.961787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b200286-19fe-4ba0-bb8a-a41fb34ae8be admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:56.967372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb21ec1-5658-4b64-930c-6e89071cfa1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:56.968016 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb21ec1-5658-4b64-930c-6e89071cfa1f admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:56.968330 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb21ec1-5658-4b64-930c-6e89071cfa1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:56.968604 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb21ec1-5658-4b64-930c-6e89071cfa1f admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:56.969256 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb21ec1-5658-4b64-930c-6e89071cfa1f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:56.971181 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb21ec1-5658-4b64-930c-6e89071cfa1f admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.001589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 788/1580] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:24:56 2022] PUT /identity/v3/projects/9774c354b8b048768f2071acfb48a4bf/users/d84b6d05decb47c7b55ac782d5839c1a/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:24:57.002559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a84842-92b3-4e6d-9935-b955f3dd0f47 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.054927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 793/1581] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:24:56 2022] POST /identity/v3/projects => generated 363 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:57.056037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb21ec1-5658-4b64-930c-6e89071cfa1f admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.073809 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a84842-92b3-4e6d-9935-b955f3dd0f47 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.081558 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:24:57.126968 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb21ec1-5658-4b64-930c-6e89071cfa1f admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.131617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2cf692-3d34-4016-9f56-35b9a5129ec7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.132254 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2cf692-3d34-4016-9f56-35b9a5129ec7 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.132525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2cf692-3d34-4016-9f56-35b9a5129ec7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.132793 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2cf692-3d34-4016-9f56-35b9a5129ec7 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.133409 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2cf692-3d34-4016-9f56-35b9a5129ec7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.135326 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2cf692-3d34-4016-9f56-35b9a5129ec7 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.136307 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f727ed7c-49c9-4d57-b16d-ffd926296567 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.137045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f727ed7c-49c9-4d57-b16d-ffd926296567 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.137331 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f727ed7c-49c9-4d57-b16d-ffd926296567 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.137588 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f727ed7c-49c9-4d57-b16d-ffd926296567 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.138344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f727ed7c-49c9-4d57-b16d-ffd926296567 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.140566 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f727ed7c-49c9-4d57-b16d-ffd926296567 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.239142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 794/1582] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:24:57 2022] POST /identity/v3/users => generated 289 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:57.239142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2cf692-3d34-4016-9f56-35b9a5129ec7 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.242957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:24:57.246658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2cf692-3d34-4016-9f56-35b9a5129ec7 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.251285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ff4733-3931-4387-aa1f-d0df35d85f23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.251869 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ff4733-3931-4387-aa1f-d0df35d85f23 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.252246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ff4733-3931-4387-aa1f-d0df35d85f23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.252529 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ff4733-3931-4387-aa1f-d0df35d85f23 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.253139 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ff4733-3931-4387-aa1f-d0df35d85f23 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.255052 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ff4733-3931-4387-aa1f-d0df35d85f23 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.274505 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 789/1583] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:24:57 2022] POST /identity/v3/users => generated 289 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:57.276292 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f727ed7c-49c9-4d57-b16d-ffd926296567 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.286060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f727ed7c-49c9-4d57-b16d-ffd926296567 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.290767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b4832e-e4ad-417c-9925-9eb7295139a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.291756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b4832e-e4ad-417c-9925-9eb7295139a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.292034 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b4832e-e4ad-417c-9925-9eb7295139a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.292344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b4832e-e4ad-417c-9925-9eb7295139a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.292969 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b4832e-e4ad-417c-9925-9eb7295139a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.294546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b4832e-e4ad-417c-9925-9eb7295139a7 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.304569 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 790/1584] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:57.305353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b4832e-e4ad-417c-9925-9eb7295139a7 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.314814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b4832e-e4ad-417c-9925-9eb7295139a7 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.319405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9ef332-1ce0-42f8-8b4d-f720d4b52d3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.319971 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9ef332-1ce0-42f8-8b4d-f720d4b52d3d admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.320544 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9ef332-1ce0-42f8-8b4d-f720d4b52d3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.320803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9ef332-1ce0-42f8-8b4d-f720d4b52d3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.321427 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9ef332-1ce0-42f8-8b4d-f720d4b52d3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.323650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9ef332-1ce0-42f8-8b4d-f720d4b52d3d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.334177 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 791/1585] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:57 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:57.337248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5acec4-b9c6-4ad7-bfd7-48da1360968f None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.337248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5acec4-b9c6-4ad7-bfd7-48da1360968f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.337248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5acec4-b9c6-4ad7-bfd7-48da1360968f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.345342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 795/1586] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:24:57 2022] POST /identity/v3/users => generated 295 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:57.347423 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06052969-4564-427f-9f97-0d9fb27dd562 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.347625 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06052969-4564-427f-9f97-0d9fb27dd562 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.348055 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06052969-4564-427f-9f97-0d9fb27dd562 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.368545 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-06052969-4564-427f-9f97-0d9fb27dd562 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:57.371748 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-df5acec4-b9c6-4ad7-bfd7-48da1360968f None None] MFA Rules not processed for user `dc3e6a6484c74d2cab56e6696404817d`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:57.371890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-06052969-4564-427f-9f97-0d9fb27dd562 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:57.371998 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-df5acec4-b9c6-4ad7-bfd7-48da1360968f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:57.377677 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 796/1587] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:57.378581 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06052969-4564-427f-9f97-0d9fb27dd562 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.389063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06052969-4564-427f-9f97-0d9fb27dd562 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.393631 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972edeea-2c8b-46e1-98c2-343ef85d418c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.394215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972edeea-2c8b-46e1-98c2-343ef85d418c admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.394498 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972edeea-2c8b-46e1-98c2-343ef85d418c admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.394769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972edeea-2c8b-46e1-98c2-343ef85d418c admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.395386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972edeea-2c8b-46e1-98c2-343ef85d418c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.397343 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972edeea-2c8b-46e1-98c2-343ef85d418c admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.441478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 792/1588] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:57 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:57.442465 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5acec4-b9c6-4ad7-bfd7-48da1360968f None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.453428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5acec4-b9c6-4ad7-bfd7-48da1360968f None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.458750 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba118cbb-e004-4301-9a32-351140311e8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.459352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba118cbb-e004-4301-9a32-351140311e8c admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.459628 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba118cbb-e004-4301-9a32-351140311e8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.459898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba118cbb-e004-4301-9a32-351140311e8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.460581 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba118cbb-e004-4301-9a32-351140311e8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.462498 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba118cbb-e004-4301-9a32-351140311e8c admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.473918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 793/1589] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:57 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:57.474989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba118cbb-e004-4301-9a32-351140311e8c admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.485849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba118cbb-e004-4301-9a32-351140311e8c admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.490771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0b508d-057b-4e10-8669-01fd3fdd0725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.491350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0b508d-057b-4e10-8669-01fd3fdd0725 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.491623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0b508d-057b-4e10-8669-01fd3fdd0725 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.491892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0b508d-057b-4e10-8669-01fd3fdd0725 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.492590 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0b508d-057b-4e10-8669-01fd3fdd0725 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.494278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0b508d-057b-4e10-8669-01fd3fdd0725 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.504670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 794/1590] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:57.505537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0b508d-057b-4e10-8669-01fd3fdd0725 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.516354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 797/1591] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:24:57 2022] POST /identity/v3/users => generated 282 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:57.516354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972edeea-2c8b-46e1-98c2-343ef85d418c admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.516750 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0b508d-057b-4e10-8669-01fd3fdd0725 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.521492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a816c8f-33ab-4a84-b0cf-da514552598c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.522091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a816c8f-33ab-4a84-b0cf-da514552598c admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.522363 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a816c8f-33ab-4a84-b0cf-da514552598c admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.522682 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a816c8f-33ab-4a84-b0cf-da514552598c admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.523317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a816c8f-33ab-4a84-b0cf-da514552598c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.524864 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972edeea-2c8b-46e1-98c2-343ef85d418c admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.524988 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a816c8f-33ab-4a84-b0cf-da514552598c admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.532630 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fff3c3-6357-4cd6-a6ff-6a8ec5612971 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.533395 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fff3c3-6357-4cd6-a6ff-6a8ec5612971 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.533676 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fff3c3-6357-4cd6-a6ff-6a8ec5612971 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.533975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fff3c3-6357-4cd6-a6ff-6a8ec5612971 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.534622 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fff3c3-6357-4cd6-a6ff-6a8ec5612971 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.536300 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fff3c3-6357-4cd6-a6ff-6a8ec5612971 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.537744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 795/1592] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:57 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:57.538688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a816c8f-33ab-4a84-b0cf-da514552598c admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.549186 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a816c8f-33ab-4a84-b0cf-da514552598c admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.560396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 798/1593] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:57 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:57.560396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fff3c3-6357-4cd6-a6ff-6a8ec5612971 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.560624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6982a6c9-54a6-406e-b8e2-bb3912573c8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.560624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6982a6c9-54a6-406e-b8e2-bb3912573c8a admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.560624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6982a6c9-54a6-406e-b8e2-bb3912573c8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.560624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6982a6c9-54a6-406e-b8e2-bb3912573c8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.561017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6982a6c9-54a6-406e-b8e2-bb3912573c8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.561017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6982a6c9-54a6-406e-b8e2-bb3912573c8a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.574192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 796/1594] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:57 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:57.574192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60d1ba4-b63b-4b39-84ea-b05e21ad81cc None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.574192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60d1ba4-b63b-4b39-84ea-b05e21ad81cc None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.574192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60d1ba4-b63b-4b39-84ea-b05e21ad81cc None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.574619 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 797/1595] 10.222.0.47 () {54 vars in 962 bytes} [Sun Aug 21 01:24:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:24:57.575077 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60d1ba4-b63b-4b39-84ea-b05e21ad81cc None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.593461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60d1ba4-b63b-4b39-84ea-b05e21ad81cc None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.597636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9592cb80-d150-402c-aa8d-0c3a19bc228f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.600301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9592cb80-d150-402c-aa8d-0c3a19bc228f admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.600301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9592cb80-d150-402c-aa8d-0c3a19bc228f admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.600301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9592cb80-d150-402c-aa8d-0c3a19bc228f admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.600301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9592cb80-d150-402c-aa8d-0c3a19bc228f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.601787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9592cb80-d150-402c-aa8d-0c3a19bc228f admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.612692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 798/1596] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:57 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:57.612886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fff3c3-6357-4cd6-a6ff-6a8ec5612971 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.613326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9592cb80-d150-402c-aa8d-0c3a19bc228f admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.620031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01acbdb1-615e-4604-9c60-6d2c215864d9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.620701 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01acbdb1-615e-4604-9c60-6d2c215864d9 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.620978 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01acbdb1-615e-4604-9c60-6d2c215864d9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.621249 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01acbdb1-615e-4604-9c60-6d2c215864d9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.623834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9592cb80-d150-402c-aa8d-0c3a19bc228f admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.628498 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0e6232-cac4-42a2-9860-7cb8beba780b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.629038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0e6232-cac4-42a2-9860-7cb8beba780b admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.629318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0e6232-cac4-42a2-9860-7cb8beba780b admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.629592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0e6232-cac4-42a2-9860-7cb8beba780b admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.630228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0e6232-cac4-42a2-9860-7cb8beba780b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.631207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01acbdb1-615e-4604-9c60-6d2c215864d9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.631810 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0e6232-cac4-42a2-9860-7cb8beba780b admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.632744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01acbdb1-615e-4604-9c60-6d2c215864d9 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.638453 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:24:57.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 799/1597] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:24:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:24:57.644222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01acbdb1-615e-4604-9c60-6d2c215864d9 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.644341 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 799/1598] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:57.645135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0e6232-cac4-42a2-9860-7cb8beba780b admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.654611 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01acbdb1-615e-4604-9c60-6d2c215864d9 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.657332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0e6232-cac4-42a2-9860-7cb8beba780b admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.661441 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac06190-be0a-488b-a103-eb8246bde943 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.674997 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac06190-be0a-488b-a103-eb8246bde943 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.674997 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac06190-be0a-488b-a103-eb8246bde943 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.674997 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac06190-be0a-488b-a103-eb8246bde943 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.675296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d9b67c-5fdd-4f58-a521-e654487503e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.675410 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac06190-be0a-488b-a103-eb8246bde943 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.675500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d9b67c-5fdd-4f58-a521-e654487503e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.675500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d9b67c-5fdd-4f58-a521-e654487503e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.675500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d9b67c-5fdd-4f58-a521-e654487503e1 admin admin] PATH_INFO: `/v3/projects/f4a0fcea08b248fe84012875a7243761/users/0cd8d822f15943a78575c3744692cdb9/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.675712 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac06190-be0a-488b-a103-eb8246bde943 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.677710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 800/1599] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:57 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:57.678340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac06190-be0a-488b-a103-eb8246bde943 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.680390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d9b67c-5fdd-4f58-a521-e654487503e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f4a0fcea08b248fe84012875a7243761, user_id=0cd8d822f15943a78575c3744692cdb9, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.683966 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d9b67c-5fdd-4f58-a521-e654487503e1 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.690532 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac06190-be0a-488b-a103-eb8246bde943 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.695293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a15eee-c71a-4c78-b6a5-5fdd6ba1b3dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.695979 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a15eee-c71a-4c78-b6a5-5fdd6ba1b3dc admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.696300 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a15eee-c71a-4c78-b6a5-5fdd6ba1b3dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.696572 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a15eee-c71a-4c78-b6a5-5fdd6ba1b3dc admin admin] PATH_INFO: `/v3/projects/0cdbd53cdbbf4cc3a1bc436bd329b60f/users/a5c6871a2f0046aeb69b82e0c1529376/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.712965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a15eee-c71a-4c78-b6a5-5fdd6ba1b3dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=0cdbd53cdbbf4cc3a1bc436bd329b60f, user_id=a5c6871a2f0046aeb69b82e0c1529376, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.715619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a15eee-c71a-4c78-b6a5-5fdd6ba1b3dc admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.807545 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 800/1600] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:24:57 2022] PUT /identity/v3/projects/f4a0fcea08b248fe84012875a7243761/users/0cd8d822f15943a78575c3744692cdb9/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:24:57.807764 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 801/1601] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:24:57 2022] PUT /identity/v3/projects/0cdbd53cdbbf4cc3a1bc436bd329b60f/users/a5c6871a2f0046aeb69b82e0c1529376/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:24:57.807908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d9b67c-5fdd-4f58-a521-e654487503e1 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.808012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2512d6-ef84-44b4-a981-9caa75f7c84a None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.808012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2512d6-ef84-44b4-a981-9caa75f7c84a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.808012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2512d6-ef84-44b4-a981-9caa75f7c84a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.861483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-aa2512d6-ef84-44b4-a981-9caa75f7c84a None None] MFA Rules not processed for user `fabdcdc25d1d4ef6bc66bd6cadecaaf8`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:57.862715 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-aa2512d6-ef84-44b4-a981-9caa75f7c84a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:57.879691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d9b67c-5fdd-4f58-a521-e654487503e1 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.885357 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ceb017-4439-4f8e-9a6b-69fca850b01e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.885962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ceb017-4439-4f8e-9a6b-69fca850b01e admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.886357 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ceb017-4439-4f8e-9a6b-69fca850b01e admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.886639 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ceb017-4439-4f8e-9a6b-69fca850b01e admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.887281 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ceb017-4439-4f8e-9a6b-69fca850b01e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.888935 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ceb017-4439-4f8e-9a6b-69fca850b01e admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.899202 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 801/1602] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:57 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:57.901329 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ceb017-4439-4f8e-9a6b-69fca850b01e admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.910044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ceb017-4439-4f8e-9a6b-69fca850b01e admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.914932 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad65374-de32-4663-809f-82c6a2798120 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.916372 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad65374-de32-4663-809f-82c6a2798120 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.916828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad65374-de32-4663-809f-82c6a2798120 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.917019 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad65374-de32-4663-809f-82c6a2798120 admin admin] PATH_INFO: `/v3/projects/a02859127960469585afd76d1bf8f6a1/users/e9b2b58261654b67a56e2d0a14adf972/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.935148 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:24:57.941696 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad65374-de32-4663-809f-82c6a2798120 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a02859127960469585afd76d1bf8f6a1, user_id=e9b2b58261654b67a56e2d0a14adf972, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.941859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad65374-de32-4663-809f-82c6a2798120 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:57.943681 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 802/1603] 10.222.0.47 () {58 vars in 1059 bytes} [Sun Aug 21 01:24:57 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:57.944799 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2512d6-ef84-44b4-a981-9caa75f7c84a None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:57.955824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2512d6-ef84-44b4-a981-9caa75f7c84a None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:57.964999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a25a0d9-3d8c-4ca1-9a1c-5ae17faca435 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:57.965679 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a25a0d9-3d8c-4ca1-9a1c-5ae17faca435 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:57.965999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a25a0d9-3d8c-4ca1-9a1c-5ae17faca435 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:57.966296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a25a0d9-3d8c-4ca1-9a1c-5ae17faca435 admin admin] PATH_INFO: `/v3/projects/a1914f8232864af6ad97aebcb1636125/users/bc7137d0ba634b6ab4b83de1ab47c476/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:57.995525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a25a0d9-3d8c-4ca1-9a1c-5ae17faca435 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1914f8232864af6ad97aebcb1636125, user_id=bc7137d0ba634b6ab4b83de1ab47c476, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:57.995753 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a25a0d9-3d8c-4ca1-9a1c-5ae17faca435 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:58.097781 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 802/1604] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:24:57 2022] PUT /identity/v3/projects/a02859127960469585afd76d1bf8f6a1/users/e9b2b58261654b67a56e2d0a14adf972/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:24:58.098844 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad65374-de32-4663-809f-82c6a2798120 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:58.128862 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 803/1605] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:24:57 2022] PUT /identity/v3/projects/a1914f8232864af6ad97aebcb1636125/users/bc7137d0ba634b6ab4b83de1ab47c476/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:24:58.130731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01631a6-9dbb-40a9-833d-68cfe0ed2617 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:58.131222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01631a6-9dbb-40a9-833d-68cfe0ed2617 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:58.131758 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01631a6-9dbb-40a9-833d-68cfe0ed2617 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:58.159906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-b01631a6-9dbb-40a9-833d-68cfe0ed2617 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:58.161320 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b01631a6-9dbb-40a9-833d-68cfe0ed2617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:58.187731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad65374-de32-4663-809f-82c6a2798120 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:58.245577 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 804/1606] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:58.249846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9e6ffd-a8e8-40e6-807d-bdc4dfff480f None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:58.250142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9e6ffd-a8e8-40e6-807d-bdc4dfff480f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:58.250451 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9e6ffd-a8e8-40e6-807d-bdc4dfff480f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:58.267385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:24:58.276877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94441002-e728-4854-ad2b-78755360ce23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:58.286124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94441002-e728-4854-ad2b-78755360ce23 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:58.286124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94441002-e728-4854-ad2b-78755360ce23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:58.286124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94441002-e728-4854-ad2b-78755360ce23 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:58.286124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94441002-e728-4854-ad2b-78755360ce23 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:58.286124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94441002-e728-4854-ad2b-78755360ce23 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:58.287783 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-5f9e6ffd-a8e8-40e6-807d-bdc4dfff480f None None] MFA Rules not processed for user `a5c6871a2f0046aeb69b82e0c1529376`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:58.289200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5f9e6ffd-a8e8-40e6-807d-bdc4dfff480f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:58.297876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 803/1607] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:58 2022] GET /identity/v3/roles => generated 1420 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:58.297876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94441002-e728-4854-ad2b-78755360ce23 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:58.307472 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94441002-e728-4854-ad2b-78755360ce23 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:58.311914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d305cac-5c33-4be6-9344-c6911aae91a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:58.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d305cac-5c33-4be6-9344-c6911aae91a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:58.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d305cac-5c33-4be6-9344-c6911aae91a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:58.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d305cac-5c33-4be6-9344-c6911aae91a5 admin admin] PATH_INFO: `/v3/projects/13b534c485db48d8a37e024702f87092/users/0f4a35c46e4741c1b238f24d203b416d/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:58.335056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d305cac-5c33-4be6-9344-c6911aae91a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=13b534c485db48d8a37e024702f87092, user_id=0f4a35c46e4741c1b238f24d203b416d, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:58.339154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d305cac-5c33-4be6-9344-c6911aae91a5 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:58.378572 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 805/1608] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:58 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:58.379463 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9e6ffd-a8e8-40e6-807d-bdc4dfff480f None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:58.440414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 804/1609] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:24:58 2022] PUT /identity/v3/projects/13b534c485db48d8a37e024702f87092/users/0f4a35c46e4741c1b238f24d203b416d/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:24:58.445299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba233687-9e6b-4014-a336-97faf6e3b3e1 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:58.445509 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba233687-9e6b-4014-a336-97faf6e3b3e1 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:58.445949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba233687-9e6b-4014-a336-97faf6e3b3e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:58.459534 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:24:58.467266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9e6ffd-a8e8-40e6-807d-bdc4dfff480f None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:58.480605 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-ba233687-9e6b-4014-a336-97faf6e3b3e1 None None] MFA Rules not processed for user `e9b2b58261654b67a56e2d0a14adf972`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:58.481182 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ba233687-9e6b-4014-a336-97faf6e3b3e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:58.546031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1a0d6e-1fc0-4483-901e-b807d6f0c8fb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:58.546592 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a0d6e-1fc0-4483-901e-b807d6f0c8fb service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:58.546914 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a0d6e-1fc0-4483-901e-b807d6f0c8fb service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:58.547154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a0d6e-1fc0-4483-901e-b807d6f0c8fb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:58.562885 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 805/1610] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:58 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:58.564291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c4a62-f188-4da4-b410-bb4d65843d85 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:58.564564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c4a62-f188-4da4-b410-bb4d65843d85 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:58.564843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c4a62-f188-4da4-b410-bb4d65843d85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:58.588369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-fc2c4a62-f188-4da4-b410-bb4d65843d85 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:58.589603 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-fc2c4a62-f188-4da4-b410-bb4d65843d85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:58.609336 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1a0d6e-1fc0-4483-901e-b807d6f0c8fb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:58.611078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1a0d6e-1fc0-4483-901e-b807d6f0c8fb service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:58.628236 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 806/1611] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:24:58 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 249 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:24:58.631858 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1a0d6e-1fc0-4483-901e-b807d6f0c8fb service nova] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:58.663524 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 806/1612] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:58.667676 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2c4a62-f188-4da4-b410-bb4d65843d85 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:58.677934 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2c4a62-f188-4da4-b410-bb4d65843d85 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:58.684503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9974986-e0a2-4e10-8a9b-a386bd943b79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:58.685812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9974986-e0a2-4e10-8a9b-a386bd943b79 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:58.686221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9974986-e0a2-4e10-8a9b-a386bd943b79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:58.686951 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9974986-e0a2-4e10-8a9b-a386bd943b79 admin admin] PATH_INFO: `/v3/projects/6df4d952c8ac4683a07161bc62dbdba5/users/b1965136d19e4fb4b13c9fd060c77635/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:58.705843 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1a0d6e-1fc0-4483-901e-b807d6f0c8fb service nova] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:58.707124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9974986-e0a2-4e10-8a9b-a386bd943b79 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6df4d952c8ac4683a07161bc62dbdba5, user_id=b1965136d19e4fb4b13c9fd060c77635, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:58.710121 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9974986-e0a2-4e10-8a9b-a386bd943b79 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:58.710462 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac189925-5f37-4037-991c-e817e4b10660 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:58.711298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac189925-5f37-4037-991c-e817e4b10660 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:58.711764 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac189925-5f37-4037-991c-e817e4b10660 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:58.712309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac189925-5f37-4037-991c-e817e4b10660 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:58.729613 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac189925-5f37-4037-991c-e817e4b10660 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:58.731828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac189925-5f37-4037-991c-e817e4b10660 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:58.742797 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 807/1613] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:24:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:24:58.744643 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e180aa30-d21b-4f3c-93bd-62f2e128c02e None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:58.744928 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e180aa30-d21b-4f3c-93bd-62f2e128c02e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:58.745199 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e180aa30-d21b-4f3c-93bd-62f2e128c02e None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:58.746602 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 808/1614] 10.222.0.47 () {54 vars in 962 bytes} [Sun Aug 21 01:24:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:24:58.747472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f25ae3-ea3b-4972-a1b6-adaa725c62f0 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:58.747754 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f25ae3-ea3b-4972-a1b6-adaa725c62f0 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:58.748026 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f25ae3-ea3b-4972-a1b6-adaa725c62f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:58.768954 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-82f25ae3-ea3b-4972-a1b6-adaa725c62f0 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:58.770181 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-82f25ae3-ea3b-4972-a1b6-adaa725c62f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:58.778826 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 809/1615] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:58.782830 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f25ae3-ea3b-4972-a1b6-adaa725c62f0 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:58.793212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f25ae3-ea3b-4972-a1b6-adaa725c62f0 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:58.798031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c59a1a-ceae-4e56-a331-041aeb4a57af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:58.798578 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c59a1a-ceae-4e56-a331-041aeb4a57af admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:58.798852 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c59a1a-ceae-4e56-a331-041aeb4a57af admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:58.799121 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c59a1a-ceae-4e56-a331-041aeb4a57af admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:58.799742 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c59a1a-ceae-4e56-a331-041aeb4a57af admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:58.801664 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c59a1a-ceae-4e56-a331-041aeb4a57af admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:58.821513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 807/1616] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:24:58 2022] PUT /identity/v3/projects/6df4d952c8ac4683a07161bc62dbdba5/users/b1965136d19e4fb4b13c9fd060c77635/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:24:58.823156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9974986-e0a2-4e10-8a9b-a386bd943b79 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:58.882629 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:24:58.908369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9974986-e0a2-4e10-8a9b-a386bd943b79 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:58.912395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e51a9a-06a5-4aa5-8c5b-9761e5325693 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:58.913007 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e51a9a-06a5-4aa5-8c5b-9761e5325693 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:58.913280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e51a9a-06a5-4aa5-8c5b-9761e5325693 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:58.913548 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e51a9a-06a5-4aa5-8c5b-9761e5325693 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:58.929548 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 810/1617] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:24:58 2022] POST /identity/v3/projects => generated 375 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:58.931437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ea1aac-06bd-4424-827f-71dd64edcf05 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:58.931437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ea1aac-06bd-4424-827f-71dd64edcf05 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:58.931733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ea1aac-06bd-4424-827f-71dd64edcf05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:58.954305 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-c8ea1aac-06bd-4424-827f-71dd64edcf05 None None] MFA Rules not processed for user `fabdcdc25d1d4ef6bc66bd6cadecaaf8`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:58.955839 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-c8ea1aac-06bd-4424-827f-71dd64edcf05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:58.982174 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:24:59.003025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e51a9a-06a5-4aa5-8c5b-9761e5325693 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:59.004625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e51a9a-06a5-4aa5-8c5b-9761e5325693 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:59.017456 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 808/1618] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:24:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:24:59.018309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e51a9a-06a5-4aa5-8c5b-9761e5325693 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:59.028742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e51a9a-06a5-4aa5-8c5b-9761e5325693 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:59.049581 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 811/1619] 10.222.0.47 () {58 vars in 1059 bytes} [Sun Aug 21 01:24:58 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:59.051311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01e7fe7-83ac-463b-ba6b-4eb11be76565 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:59.051677 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01e7fe7-83ac-463b-ba6b-4eb11be76565 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:59.052023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01e7fe7-83ac-463b-ba6b-4eb11be76565 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:59.083693 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f01e7fe7-83ac-463b-ba6b-4eb11be76565 None None] MFA Rules not processed for user `b1965136d19e4fb4b13c9fd060c77635`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:59.085332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f01e7fe7-83ac-463b-ba6b-4eb11be76565 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:59.098793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18a405f-e1d1-451b-900a-44f1961b0bdb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:59.099413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18a405f-e1d1-451b-900a-44f1961b0bdb service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:59.099691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18a405f-e1d1-451b-900a-44f1961b0bdb service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:59.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18a405f-e1d1-451b-900a-44f1961b0bdb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:59.186241 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18a405f-e1d1-451b-900a-44f1961b0bdb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:59.188091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18a405f-e1d1-451b-900a-44f1961b0bdb service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:59.204242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 809/1620] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:24:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:24:59.206000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e9b1ba-b634-4817-9dae-9feca2ec153d None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:59.206343 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e9b1ba-b634-4817-9dae-9feca2ec153d None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:59.206569 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e9b1ba-b634-4817-9dae-9feca2ec153d None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:59.208328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 810/1621] 10.222.0.47 () {54 vars in 912 bytes} [Sun Aug 21 01:24:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:24:59.214858 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e9b1ba-b634-4817-9dae-9feca2ec153d None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:59.223166 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 812/1622] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:24:59 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:59.227405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01e7fe7-83ac-463b-ba6b-4eb11be76565 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:59.233005 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e9b1ba-b634-4817-9dae-9feca2ec153d None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:59.238884 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d68b2f5-1b99-4da7-89eb-5c11a4d043f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:59.239654 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d68b2f5-1b99-4da7-89eb-5c11a4d043f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:59.239933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d68b2f5-1b99-4da7-89eb-5c11a4d043f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:59.240352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d68b2f5-1b99-4da7-89eb-5c11a4d043f7 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:59.240959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d68b2f5-1b99-4da7-89eb-5c11a4d043f7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:59.245001 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d68b2f5-1b99-4da7-89eb-5c11a4d043f7 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:59.251321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01e7fe7-83ac-463b-ba6b-4eb11be76565 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:59.263882 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115cacf8-4e77-4f96-b853-6e859e10b208 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:59.265423 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115cacf8-4e77-4f96-b853-6e859e10b208 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:59.266035 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115cacf8-4e77-4f96-b853-6e859e10b208 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:59.266599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115cacf8-4e77-4f96-b853-6e859e10b208 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:59.363615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 811/1623] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:24:59 2022] POST /identity/v3/users => generated 294 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:59.365845 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e95f1a5-8f74-4fc0-af3d-399e634140a7 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:59.365845 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e95f1a5-8f74-4fc0-af3d-399e634140a7 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:59.365845 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e95f1a5-8f74-4fc0-af3d-399e634140a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:59.412549 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-6e95f1a5-8f74-4fc0-af3d-399e634140a7 None None] MFA Rules not processed for user `fabdcdc25d1d4ef6bc66bd6cadecaaf8`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:24:59.414714 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6e95f1a5-8f74-4fc0-af3d-399e634140a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:24:59.425008 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115cacf8-4e77-4f96-b853-6e859e10b208 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:59.430962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115cacf8-4e77-4f96-b853-6e859e10b208 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:59.435700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 812/1624] 10.222.0.47 () {58 vars in 1009 bytes} [Sun Aug 21 01:24:59 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:24:59.436813 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e95f1a5-8f74-4fc0-af3d-399e634140a7 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:59.453094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e95f1a5-8f74-4fc0-af3d-399e634140a7 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:59.458729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ee9d09-16fb-4267-a7cd-f02885d1f385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:59.465218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ee9d09-16fb-4267-a7cd-f02885d1f385 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:59.465218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ee9d09-16fb-4267-a7cd-f02885d1f385 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:59.474532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ee9d09-16fb-4267-a7cd-f02885d1f385 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:59.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ee9d09-16fb-4267-a7cd-f02885d1f385 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:59.478165 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 813/1625] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:24:59 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 254 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:24:59.479780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115cacf8-4e77-4f96-b853-6e859e10b208 service nova] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:59.489373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ee9d09-16fb-4267-a7cd-f02885d1f385 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:59.508213 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115cacf8-4e77-4f96-b853-6e859e10b208 service nova] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:59.510727 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:24:59.516705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ad3b0f-450f-407a-96ec-cc43e7653c95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:59.517803 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ad3b0f-450f-407a-96ec-cc43e7653c95 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:59.517803 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ad3b0f-450f-407a-96ec-cc43e7653c95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:59.518601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ad3b0f-450f-407a-96ec-cc43e7653c95 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:59.520678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ad3b0f-450f-407a-96ec-cc43e7653c95 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:59.521820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ad3b0f-450f-407a-96ec-cc43e7653c95 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:59.545418 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 814/1626] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:24:59 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:24:59.545418 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ad3b0f-450f-407a-96ec-cc43e7653c95 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:59.615344 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:24:59.637710 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 813/1627] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:24:59 2022] POST /identity/v3/projects => generated 365 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:24:59.637710 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ee9d09-16fb-4267-a7cd-f02885d1f385 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:59.680947 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ad3b0f-450f-407a-96ec-cc43e7653c95 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:59.764373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ee9d09-16fb-4267-a7cd-f02885d1f385 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:59.771962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c2bfe6-d42b-4763-a4d8-ad028a0ecbae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:59.773539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c2bfe6-d42b-4763-a4d8-ad028a0ecbae admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:59.774225 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c2bfe6-d42b-4763-a4d8-ad028a0ecbae admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:59.775563 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c2bfe6-d42b-4763-a4d8-ad028a0ecbae admin admin] PATH_INFO: `/v3/projects/e0c09ae9228744f4908068a08ee77e35/users/2401a6bcd73f44f0898ece573a0cb180/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:59.801331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7dab116-5b12-40fa-b49b-3f65cd6796fb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:24:59.802247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dab116-5b12-40fa-b49b-3f65cd6796fb service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:24:59.802625 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dab116-5b12-40fa-b49b-3f65cd6796fb service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:24:59.802929 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dab116-5b12-40fa-b49b-3f65cd6796fb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:24:59.826466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c2bfe6-d42b-4763-a4d8-ad028a0ecbae admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0c09ae9228744f4908068a08ee77e35, user_id=2401a6bcd73f44f0898ece573a0cb180, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:59.837425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c2bfe6-d42b-4763-a4d8-ad028a0ecbae admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:59.927415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7dab116-5b12-40fa-b49b-3f65cd6796fb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:24:59.933672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7dab116-5b12-40fa-b49b-3f65cd6796fb service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:24:59.958880 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 815/1628] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:24:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:24:59.960454 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7dab116-5b12-40fa-b49b-3f65cd6796fb service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:24:59.974818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7dab116-5b12-40fa-b49b-3f65cd6796fb service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:24:59.986370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 814/1629] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:24:59 2022] PUT /identity/v3/projects/e0c09ae9228744f4908068a08ee77e35/users/2401a6bcd73f44f0898ece573a0cb180/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:24:59.998711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bb9e66-7604-46b0-bde8-376349825479 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:00.001060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bb9e66-7604-46b0-bde8-376349825479 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:00.001542 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bb9e66-7604-46b0-bde8-376349825479 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:00.007759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 815/1630] 10.222.0.47 () {54 vars in 912 bytes} [Sun Aug 21 01:24:59 2022] GET /identity => generated 270 bytes in 21 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:25:00.012309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bb9e66-7604-46b0-bde8-376349825479 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:00.111133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5537f9-a5b3-49f3-8cef-68ef73dee0e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:00.111765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5537f9-a5b3-49f3-8cef-68ef73dee0e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:00.112048 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5537f9-a5b3-49f3-8cef-68ef73dee0e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:00.112338 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5537f9-a5b3-49f3-8cef-68ef73dee0e6 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:00.114534 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5537f9-a5b3-49f3-8cef-68ef73dee0e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:00.119278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5537f9-a5b3-49f3-8cef-68ef73dee0e6 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:00.206670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bb9e66-7604-46b0-bde8-376349825479 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:00.213179 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4147f1-a404-495c-9dde-6fc105f2602a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:00.214743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4147f1-a404-495c-9dde-6fc105f2602a admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:00.215338 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4147f1-a404-495c-9dde-6fc105f2602a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:00.216320 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4147f1-a404-495c-9dde-6fc105f2602a admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:00.217435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4147f1-a404-495c-9dde-6fc105f2602a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:00.220563 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4147f1-a404-495c-9dde-6fc105f2602a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:00.248784 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 816/1631] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:00 2022] GET /identity/v3/roles => generated 1420 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:00.250579 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdca856-aa4b-4c0f-a1e1-0560b5a44a2c None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:00.251092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdca856-aa4b-4c0f-a1e1-0560b5a44a2c None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:00.251701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdca856-aa4b-4c0f-a1e1-0560b5a44a2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:00.284005 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 816/1632] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:24:59 2022] POST /identity/v3/users => generated 289 bytes in 324 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:00.286509 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5537f9-a5b3-49f3-8cef-68ef73dee0e6 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:00.402009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-2fdca856-aa4b-4c0f-a1e1-0560b5a44a2c None None] MFA Rules not processed for user `fabdcdc25d1d4ef6bc66bd6cadecaaf8`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:00.402009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-2fdca856-aa4b-4c0f-a1e1-0560b5a44a2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:00.402708 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5537f9-a5b3-49f3-8cef-68ef73dee0e6 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:00.402708 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc0d7bc-f01b-4cce-8e91-612260f49dd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:00.402708 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc0d7bc-f01b-4cce-8e91-612260f49dd6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:00.402708 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc0d7bc-f01b-4cce-8e91-612260f49dd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:00.403069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc0d7bc-f01b-4cce-8e91-612260f49dd6 admin admin] PATH_INFO: `/v3/projects/e0c09ae9228744f4908068a08ee77e35/users/2401a6bcd73f44f0898ece573a0cb180/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:00.403069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc0d7bc-f01b-4cce-8e91-612260f49dd6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0c09ae9228744f4908068a08ee77e35, user_id=2401a6bcd73f44f0898ece573a0cb180, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:00.403069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc0d7bc-f01b-4cce-8e91-612260f49dd6 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:00.403069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:00.403069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 817/1633] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:00 2022] PUT /identity/v3/projects/e0c09ae9228744f4908068a08ee77e35/users/2401a6bcd73f44f0898ece573a0cb180/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:00.405939 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc0d7bc-f01b-4cce-8e91-612260f49dd6 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:00.494328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:00.530220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 817/1634] 10.222.0.47 () {58 vars in 1009 bytes} [Sun Aug 21 01:25:00 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:00.531566 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce3b772-3cbe-4803-8f84-361f6153fe45 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:00.532222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce3b772-3cbe-4803-8f84-361f6153fe45 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:00.534224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce3b772-3cbe-4803-8f84-361f6153fe45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:00.611766 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-5ce3b772-3cbe-4803-8f84-361f6153fe45 None None] MFA Rules not processed for user `2401a6bcd73f44f0898ece573a0cb180`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:00.613643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-5ce3b772-3cbe-4803-8f84-361f6153fe45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:00.756291 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc0d7bc-f01b-4cce-8e91-612260f49dd6 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:00.778220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 818/1635] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:00 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:00.796778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6662742a-aaf4-420c-8d9b-edeed50297bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:00.796778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6662742a-aaf4-420c-8d9b-edeed50297bc admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:00.796778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6662742a-aaf4-420c-8d9b-edeed50297bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:00.796778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6662742a-aaf4-420c-8d9b-edeed50297bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:00.797496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce3b772-3cbe-4803-8f84-361f6153fe45 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:00.797600 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6662742a-aaf4-420c-8d9b-edeed50297bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:00.801539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6662742a-aaf4-420c-8d9b-edeed50297bc admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:00.822241 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 818/1636] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:00 2022] GET /identity/v3/roles => generated 1420 bytes in 424 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:00.829197 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6662742a-aaf4-420c-8d9b-edeed50297bc admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:00.847856 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6662742a-aaf4-420c-8d9b-edeed50297bc admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:00.863078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32ae56e-b17c-4f0b-aa3e-ffdfe79519c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:00.863892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32ae56e-b17c-4f0b-aa3e-ffdfe79519c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:00.864332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32ae56e-b17c-4f0b-aa3e-ffdfe79519c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:00.864694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32ae56e-b17c-4f0b-aa3e-ffdfe79519c9 admin admin] PATH_INFO: `/v3/projects/e46a7357852c41d9b794764bff4c2b41/users/26620316dced41389244d395eea2b037/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:00.898200 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce3b772-3cbe-4803-8f84-361f6153fe45 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:00.899412 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32ae56e-b17c-4f0b-aa3e-ffdfe79519c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e46a7357852c41d9b794764bff4c2b41, user_id=26620316dced41389244d395eea2b037, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:00.902291 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32ae56e-b17c-4f0b-aa3e-ffdfe79519c9 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:00.913441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2415cdf0-bd82-4acd-a541-b396f8b16cca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:00.913441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2415cdf0-bd82-4acd-a541-b396f8b16cca service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:00.913441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2415cdf0-bd82-4acd-a541-b396f8b16cca service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:00.913441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2415cdf0-bd82-4acd-a541-b396f8b16cca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:01.043038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 819/1637] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:00 2022] PUT /identity/v3/projects/e46a7357852c41d9b794764bff4c2b41/users/26620316dced41389244d395eea2b037/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:01.048094 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32ae56e-b17c-4f0b-aa3e-ffdfe79519c9 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:01.091303 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2415cdf0-bd82-4acd-a541-b396f8b16cca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:01.093390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2415cdf0-bd82-4acd-a541-b396f8b16cca service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:01.108373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 819/1638] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:01.150220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32ae56e-b17c-4f0b-aa3e-ffdfe79519c9 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:01.155010 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f20a82-c79b-4b6b-9065-c9b8ecb31951 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:01.155618 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f20a82-c79b-4b6b-9065-c9b8ecb31951 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:01.155893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f20a82-c79b-4b6b-9065-c9b8ecb31951 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:01.156198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f20a82-c79b-4b6b-9065-c9b8ecb31951 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:01.156832 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f20a82-c79b-4b6b-9065-c9b8ecb31951 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:01.158473 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f20a82-c79b-4b6b-9065-c9b8ecb31951 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:01.681349 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 820/1639] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:01 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:25:01.688829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2415cdf0-bd82-4acd-a541-b396f8b16cca service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:01.688829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2415cdf0-bd82-4acd-a541-b396f8b16cca service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:01.694616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f20a82-c79b-4b6b-9065-c9b8ecb31951 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:01.699575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f8df04-ca86-41ab-b451-8275c261fe66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:01.699575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f8df04-ca86-41ab-b451-8275c261fe66 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:01.699575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f8df04-ca86-41ab-b451-8275c261fe66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:01.699575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f8df04-ca86-41ab-b451-8275c261fe66 admin admin] PATH_INFO: `/v3/projects/e46a7357852c41d9b794764bff4c2b41/users/26620316dced41389244d395eea2b037/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:01.706199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f8df04-ca86-41ab-b451-8275c261fe66 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e46a7357852c41d9b794764bff4c2b41, user_id=26620316dced41389244d395eea2b037, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:01.706199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f8df04-ca86-41ab-b451-8275c261fe66 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:01.706199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 820/1640] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:01 2022] PUT /identity/v3/projects/e46a7357852c41d9b794764bff4c2b41/users/26620316dced41389244d395eea2b037/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:01.706199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cce6a5-6a44-4f0b-9584-ed87e5fad469 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:01.706199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cce6a5-6a44-4f0b-9584-ed87e5fad469 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:01.706199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cce6a5-6a44-4f0b-9584-ed87e5fad469 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:01.710923 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-05cce6a5-6a44-4f0b-9584-ed87e5fad469 None None] MFA Rules not processed for user `26620316dced41389244d395eea2b037`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:01.710923 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-05cce6a5-6a44-4f0b-9584-ed87e5fad469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:01.715347 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f20a82-c79b-4b6b-9065-c9b8ecb31951 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:01.716455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 821/1641] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:01 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:01.717063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a01efb-9c01-4454-8405-07b25b91982d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:01.717063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a01efb-9c01-4454-8405-07b25b91982d service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:01.717063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a01efb-9c01-4454-8405-07b25b91982d service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:01.717063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a01efb-9c01-4454-8405-07b25b91982d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:01.718437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a01efb-9c01-4454-8405-07b25b91982d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:01.718437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a01efb-9c01-4454-8405-07b25b91982d service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:01.718437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 821/1642] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:01 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:01.779101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:01.916806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cce6a5-6a44-4f0b-9584-ed87e5fad469 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:01.927120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cce6a5-6a44-4f0b-9584-ed87e5fad469 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:01.942305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9997c5b-53e1-4b61-a98a-e16802451f19 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:01.943700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9997c5b-53e1-4b61-a98a-e16802451f19 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:01.943927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9997c5b-53e1-4b61-a98a-e16802451f19 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:01.944267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9997c5b-53e1-4b61-a98a-e16802451f19 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:01.970888 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9997c5b-53e1-4b61-a98a-e16802451f19 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:01.972388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9997c5b-53e1-4b61-a98a-e16802451f19 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:01.986928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:02.006860 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 822/1643] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:01 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:02.333159 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a01efb-9c01-4454-8405-07b25b91982d service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:02.448582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a01efb-9c01-4454-8405-07b25b91982d service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:02.455662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e9d760-9cb8-4385-af24-6ce6f4c8845b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:02.457560 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e9d760-9cb8-4385-af24-6ce6f4c8845b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:02.457841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e9d760-9cb8-4385-af24-6ce6f4c8845b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:02.458113 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e9d760-9cb8-4385-af24-6ce6f4c8845b admin admin] PATH_INFO: `/v3/users/a5c6871a2f0046aeb69b82e0c1529376` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:02.459414 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e9d760-9cb8-4385-af24-6ce6f4c8845b admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5c6871a2f0046aeb69b82e0c1529376)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:02.460504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: WARNING py.warnings [None req-e0e9d760-9cb8-4385-af24-6ce6f4c8845b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 21 01:25:02.460504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: warnings.warn(msg) Aug 21 01:25:02.460504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: Aug 21 01:25:02.462053 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e9d760-9cb8-4385-af24-6ce6f4c8845b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:02.653858 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9997c5b-53e1-4b61-a98a-e16802451f19 service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:02.663602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9997c5b-53e1-4b61-a98a-e16802451f19 service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:02.685028 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6975b02b-a3e8-4fc7-bd00-2c40b4e69b3d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:02.685946 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6975b02b-a3e8-4fc7-bd00-2c40b4e69b3d service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:02.686650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6975b02b-a3e8-4fc7-bd00-2c40b4e69b3d service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:02.687113 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6975b02b-a3e8-4fc7-bd00-2c40b4e69b3d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:02.765013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6975b02b-a3e8-4fc7-bd00-2c40b4e69b3d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:02.766790 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6975b02b-a3e8-4fc7-bd00-2c40b4e69b3d service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:02.779414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 823/1644] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:02 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:02.992984 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e0e9d760-9cb8-4385-af24-6ce6f4c8845b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5c6871a2f0046aeb69b82e0c1529376'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:03.011024 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e0e9d760-9cb8-4385-af24-6ce6f4c8845b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5c6871a2f0046aeb69b82e0c1529376'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:03.013578 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e0e9d760-9cb8-4385-af24-6ce6f4c8845b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5c6871a2f0046aeb69b82e0c1529376'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:03.063317 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e0e9d760-9cb8-4385-af24-6ce6f4c8845b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5c6871a2f0046aeb69b82e0c1529376'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:03.231750 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e0e9d760-9cb8-4385-af24-6ce6f4c8845b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5c6871a2f0046aeb69b82e0c1529376'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:03.250535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 822/1645] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:02 2022] DELETE /identity/v3/users/a5c6871a2f0046aeb69b82e0c1529376 => generated 0 bytes in 915 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:03.253397 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e0c8aa-f320-4585-b2b4-89a48bf99752 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:03.254413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e0c8aa-f320-4585-b2b4-89a48bf99752 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:03.254928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e0c8aa-f320-4585-b2b4-89a48bf99752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:03.302937 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-06e0c8aa-f320-4585-b2b4-89a48bf99752 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:03.309715 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-06e0c8aa-f320-4585-b2b4-89a48bf99752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:03.452332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 824/1646] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:03.490058 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e9d760-9cb8-4385-af24-6ce6f4c8845b admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:03.491660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e0e9d760-9cb8-4385-af24-6ce6f4c8845b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:03.630874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:03.748745 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e9d760-9cb8-4385-af24-6ce6f4c8845b admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:03.758807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707e05e7-036c-4766-a256-699928d4c285 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:03.760558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707e05e7-036c-4766-a256-699928d4c285 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:03.761101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707e05e7-036c-4766-a256-699928d4c285 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:03.761736 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707e05e7-036c-4766-a256-699928d4c285 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:03.816192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707e05e7-036c-4766-a256-699928d4c285 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:03.819808 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707e05e7-036c-4766-a256-699928d4c285 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:03.869709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 823/1647] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:06.835503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e0c8aa-f320-4585-b2b4-89a48bf99752 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:06.835503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-06e0c8aa-f320-4585-b2b4-89a48bf99752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:06.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e0c8aa-f320-4585-b2b4-89a48bf99752 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:06.860797 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7ac4f4-cde5-4662-8409-b8408bb49b3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:06.862809 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7ac4f4-cde5-4662-8409-b8408bb49b3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:06.863118 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7ac4f4-cde5-4662-8409-b8408bb49b3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:06.863396 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7ac4f4-cde5-4662-8409-b8408bb49b3f admin admin] PATH_INFO: `/v3/projects/0cdbd53cdbbf4cc3a1bc436bd329b60f` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:06.873270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7ac4f4-cde5-4662-8409-b8408bb49b3f admin admin] RBAC: Authorizing `identity:delete_project(project_id=0cdbd53cdbbf4cc3a1bc436bd329b60f)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:06.874683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: WARNING py.warnings [None req-3e7ac4f4-cde5-4662-8409-b8408bb49b3f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 21 01:25:06.874683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: warnings.warn(msg) Aug 21 01:25:06.874683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: Aug 21 01:25:06.877043 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7ac4f4-cde5-4662-8409-b8408bb49b3f admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:06.932516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:07.097254 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3e7ac4f4-cde5-4662-8409-b8408bb49b3f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0cdbd53cdbbf4cc3a1bc436bd329b60f', 'request_id': 'req-3e7ac4f4-cde5-4662-8409-b8408bb49b3f'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:07.275985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3e7ac4f4-cde5-4662-8409-b8408bb49b3f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0cdbd53cdbbf4cc3a1bc436bd329b60f', 'request_id': 'req-3e7ac4f4-cde5-4662-8409-b8408bb49b3f'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:07.289263 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3e7ac4f4-cde5-4662-8409-b8408bb49b3f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0cdbd53cdbbf4cc3a1bc436bd329b60f', 'request_id': 'req-3e7ac4f4-cde5-4662-8409-b8408bb49b3f'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:07.375121 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3e7ac4f4-cde5-4662-8409-b8408bb49b3f admin admin] The token cache is being invalidate because project 0cdbd53cdbbf4cc3a1bc436bd329b60f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:07.375493 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3e7ac4f4-cde5-4662-8409-b8408bb49b3f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:07.376796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3e7ac4f4-cde5-4662-8409-b8408bb49b3f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:07.379556 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 825/1648] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:06 2022] DELETE /identity/v3/projects/0cdbd53cdbbf4cc3a1bc436bd329b60f => generated 0 bytes in 554 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:07.386060 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a991a624-1df2-4e2b-aa9e-b6514965df3a None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:07.387117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a991a624-1df2-4e2b-aa9e-b6514965df3a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:07.387624 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a991a624-1df2-4e2b-aa9e-b6514965df3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:07.460265 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-a991a624-1df2-4e2b-aa9e-b6514965df3a None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:07.461609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a991a624-1df2-4e2b-aa9e-b6514965df3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:07.681789 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 824/1649] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 298 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:07.689661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbf1756-56f0-4981-8396-78b872b7c626 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:07.689943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbf1756-56f0-4981-8396-78b872b7c626 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:07.690757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbf1756-56f0-4981-8396-78b872b7c626 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:07.722113 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-cfbf1756-56f0-4981-8396-78b872b7c626 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:07.725845 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-cfbf1756-56f0-4981-8396-78b872b7c626 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:07.767252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 826/1650] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:07.768366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a991a624-1df2-4e2b-aa9e-b6514965df3a None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:07.789352 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a991a624-1df2-4e2b-aa9e-b6514965df3a None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:07.808950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e9f287-e19c-4b0e-8b6b-fbc6d8c8012a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:07.809582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e9f287-e19c-4b0e-8b6b-fbc6d8c8012a admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:07.810020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e9f287-e19c-4b0e-8b6b-fbc6d8c8012a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:07.810301 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e9f287-e19c-4b0e-8b6b-fbc6d8c8012a admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:07.811327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e9f287-e19c-4b0e-8b6b-fbc6d8c8012a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:25:07.811633 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e9f287-e19c-4b0e-8b6b-fbc6d8c8012a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:07.813991 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e9f287-e19c-4b0e-8b6b-fbc6d8c8012a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:07.851636 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 825/1651] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:25:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:25:07.856548 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbf1756-56f0-4981-8396-78b872b7c626 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:07.887423 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbf1756-56f0-4981-8396-78b872b7c626 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:07.892194 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dd9bcd-145a-4759-b0b9-bfe41526d0cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:07.893016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dd9bcd-145a-4759-b0b9-bfe41526d0cf admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:07.893503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dd9bcd-145a-4759-b0b9-bfe41526d0cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:07.893963 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dd9bcd-145a-4759-b0b9-bfe41526d0cf admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:07.898583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dd9bcd-145a-4759-b0b9-bfe41526d0cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:07.900197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dd9bcd-145a-4759-b0b9-bfe41526d0cf admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:08.011035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 827/1652] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:07 2022] POST /identity/v3/projects => generated 369 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:08.015253 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e9f287-e19c-4b0e-8b6b-fbc6d8c8012a admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:08.038086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e9f287-e19c-4b0e-8b6b-fbc6d8c8012a admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:08.179516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ac2897-3489-46be-b16f-c311a601da38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:08.188133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ac2897-3489-46be-b16f-c311a601da38 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:08.188579 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ac2897-3489-46be-b16f-c311a601da38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:08.188854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ac2897-3489-46be-b16f-c311a601da38 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:08.189555 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ac2897-3489-46be-b16f-c311a601da38 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:08.199719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ac2897-3489-46be-b16f-c311a601da38 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:08.317589 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 826/1653] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:08 2022] POST /identity/v3/users => generated 285 bytes in 302 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:08.323786 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dd9bcd-145a-4759-b0b9-bfe41526d0cf admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:08.357373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dd9bcd-145a-4759-b0b9-bfe41526d0cf admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:08.377198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3c7d9f-417c-488e-863d-7f6c544e73fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:08.378515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3c7d9f-417c-488e-863d-7f6c544e73fc admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:08.379000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3c7d9f-417c-488e-863d-7f6c544e73fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:08.380663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3c7d9f-417c-488e-863d-7f6c544e73fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:08.381125 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3c7d9f-417c-488e-863d-7f6c544e73fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:08.384114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3c7d9f-417c-488e-863d-7f6c544e73fc admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:08.405457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 828/1654] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:08 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:08.405946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ac2897-3489-46be-b16f-c311a601da38 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:08.420747 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ac2897-3489-46be-b16f-c311a601da38 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:08.427202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd82cec6-65ad-44c5-9977-590caacb8ee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:08.428373 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd82cec6-65ad-44c5-9977-590caacb8ee1 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:08.428373 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd82cec6-65ad-44c5-9977-590caacb8ee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:08.428768 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd82cec6-65ad-44c5-9977-590caacb8ee1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:08.429697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd82cec6-65ad-44c5-9977-590caacb8ee1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:08.431694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd82cec6-65ad-44c5-9977-590caacb8ee1 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:08.453379 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3c7d9f-417c-488e-863d-7f6c544e73fc admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:08.461199 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 827/1655] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:08 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:08.466796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3c7d9f-417c-488e-863d-7f6c544e73fc admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:08.472400 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18c4c83-9818-444d-9893-a53583b8bb78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:08.473486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18c4c83-9818-444d-9893-a53583b8bb78 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:08.477333 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18c4c83-9818-444d-9893-a53583b8bb78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:08.477333 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18c4c83-9818-444d-9893-a53583b8bb78 admin admin] PATH_INFO: `/v3/projects/33ee98a5fc5743a59a10cff298eda51f/users/4017b4ef70b34e4e822005a4ca02dd73/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:08.495087 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18c4c83-9818-444d-9893-a53583b8bb78 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33ee98a5fc5743a59a10cff298eda51f, user_id=4017b4ef70b34e4e822005a4ca02dd73, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:08.498015 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18c4c83-9818-444d-9893-a53583b8bb78 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:08.582354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 829/1656] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:08 2022] PUT /identity/v3/projects/33ee98a5fc5743a59a10cff298eda51f/users/4017b4ef70b34e4e822005a4ca02dd73/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:08.593411 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0814b6-a12e-4900-8b56-3fc3f400da32 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:08.593879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0814b6-a12e-4900-8b56-3fc3f400da32 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:08.594166 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0814b6-a12e-4900-8b56-3fc3f400da32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:08.603525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:DELETE=1,keystone:SELECT=96,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:08.658374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-de0814b6-a12e-4900-8b56-3fc3f400da32 None None] MFA Rules not processed for user `4017b4ef70b34e4e822005a4ca02dd73`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:08.659867 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-de0814b6-a12e-4900-8b56-3fc3f400da32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:08.833530 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 828/1657] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:08 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:08.834704 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18c4c83-9818-444d-9893-a53583b8bb78 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:09.057758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18c4c83-9818-444d-9893-a53583b8bb78 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:09.066564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921f6a41-ee4f-4749-804a-18b11820bd1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:09.066983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921f6a41-ee4f-4749-804a-18b11820bd1a admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:09.069092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921f6a41-ee4f-4749-804a-18b11820bd1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:09.069092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921f6a41-ee4f-4749-804a-18b11820bd1a admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:09.069092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921f6a41-ee4f-4749-804a-18b11820bd1a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:09.071372 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921f6a41-ee4f-4749-804a-18b11820bd1a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:09.207033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 830/1658] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:08 2022] POST /identity/v3/projects => generated 367 bytes in 372 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:09.209974 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0814b6-a12e-4900-8b56-3fc3f400da32 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:09.228043 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0814b6-a12e-4900-8b56-3fc3f400da32 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:09.365642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6723615-b11d-461d-bf71-cb4b6d1a8186 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:09.366341 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6723615-b11d-461d-bf71-cb4b6d1a8186 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:09.366695 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6723615-b11d-461d-bf71-cb4b6d1a8186 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:09.367471 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6723615-b11d-461d-bf71-cb4b6d1a8186 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:09.370128 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6723615-b11d-461d-bf71-cb4b6d1a8186 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:09.374158 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6723615-b11d-461d-bf71-cb4b6d1a8186 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:09.473219 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 829/1659] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:09 2022] POST /identity/v3/users => generated 290 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:09.474768 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921f6a41-ee4f-4749-804a-18b11820bd1a admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:09.498023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921f6a41-ee4f-4749-804a-18b11820bd1a admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:09.514288 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a94f80e-2b3d-4a6a-964d-15d34fe52105 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:09.515028 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a94f80e-2b3d-4a6a-964d-15d34fe52105 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:09.520902 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a94f80e-2b3d-4a6a-964d-15d34fe52105 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:09.520902 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a94f80e-2b3d-4a6a-964d-15d34fe52105 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:09.520902 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a94f80e-2b3d-4a6a-964d-15d34fe52105 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:09.520902 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a94f80e-2b3d-4a6a-964d-15d34fe52105 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:09.548736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 831/1660] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:09 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:09.550306 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6723615-b11d-461d-bf71-cb4b6d1a8186 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:09.572460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6723615-b11d-461d-bf71-cb4b6d1a8186 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:09.585377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7520ed-0d22-42d9-89de-528d578e2614 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:09.589829 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7520ed-0d22-42d9-89de-528d578e2614 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:09.593440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7520ed-0d22-42d9-89de-528d578e2614 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:09.594177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7520ed-0d22-42d9-89de-528d578e2614 admin admin] PATH_INFO: `/v3/projects/7a76b10f0fd0405c93444e074e0db38b/users/6e4f494b3ea04cd7bb9a0ab22a890259/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:09.629601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7520ed-0d22-42d9-89de-528d578e2614 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a76b10f0fd0405c93444e074e0db38b, user_id=6e4f494b3ea04cd7bb9a0ab22a890259, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:09.632600 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7520ed-0d22-42d9-89de-528d578e2614 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:09.756464 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 830/1661] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:09 2022] PUT /identity/v3/projects/7a76b10f0fd0405c93444e074e0db38b/users/6e4f494b3ea04cd7bb9a0ab22a890259/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:09.756562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a94f80e-2b3d-4a6a-964d-15d34fe52105 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:09.779720 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a94f80e-2b3d-4a6a-964d-15d34fe52105 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:09.813864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:09.916397 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1feea905-6a68-4471-8c82-86a4984e607c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:09.917311 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feea905-6a68-4471-8c82-86a4984e607c admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:09.917581 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feea905-6a68-4471-8c82-86a4984e607c admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:09.917998 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feea905-6a68-4471-8c82-86a4984e607c admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:09.918737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1feea905-6a68-4471-8c82-86a4984e607c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:09.920419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1feea905-6a68-4471-8c82-86a4984e607c admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:09.933180 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 832/1662] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:09 2022] GET /identity/v3/roles => generated 1420 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:09.938133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7520ed-0d22-42d9-89de-528d578e2614 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:09.952024 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7520ed-0d22-42d9-89de-528d578e2614 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:09.962548 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f29b435-8377-44d2-9842-46d6d2e4b65a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:09.965893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f29b435-8377-44d2-9842-46d6d2e4b65a admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:09.966419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f29b435-8377-44d2-9842-46d6d2e4b65a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:09.966880 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f29b435-8377-44d2-9842-46d6d2e4b65a admin admin] PATH_INFO: `/v3/projects/7a76b10f0fd0405c93444e074e0db38b/users/6e4f494b3ea04cd7bb9a0ab22a890259/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:09.971065 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f29b435-8377-44d2-9842-46d6d2e4b65a admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a76b10f0fd0405c93444e074e0db38b, user_id=6e4f494b3ea04cd7bb9a0ab22a890259, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:09.974344 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f29b435-8377-44d2-9842-46d6d2e4b65a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:10.007382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 831/1663] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:09 2022] PUT /identity/v3/projects/7a76b10f0fd0405c93444e074e0db38b/users/6e4f494b3ea04cd7bb9a0ab22a890259/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:10.008087 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1a0ea4-40a1-4565-a92b-26640d6682ea None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:10.008424 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1a0ea4-40a1-4565-a92b-26640d6682ea None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:10.008695 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1a0ea4-40a1-4565-a92b-26640d6682ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:10.054231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-df1a0ea4-40a1-4565-a92b-26640d6682ea None None] MFA Rules not processed for user `6e4f494b3ea04cd7bb9a0ab22a890259`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:10.055771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-df1a0ea4-40a1-4565-a92b-26640d6682ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:10.194913 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 833/1664] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:10 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:25:10.203892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f29b435-8377-44d2-9842-46d6d2e4b65a admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:10.205828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-2f29b435-8377-44d2-9842-46d6d2e4b65a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:10.263311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:10.279789 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f29b435-8377-44d2-9842-46d6d2e4b65a admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:10.284192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71723b84-d45c-4cc8-84f3-30d344f544a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:10.284970 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71723b84-d45c-4cc8-84f3-30d344f544a2 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:10.285371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71723b84-d45c-4cc8-84f3-30d344f544a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:10.285740 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71723b84-d45c-4cc8-84f3-30d344f544a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:10.300056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71723b84-d45c-4cc8-84f3-30d344f544a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:10.301966 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71723b84-d45c-4cc8-84f3-30d344f544a2 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:10.314162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 832/1665] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:10 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:10.475570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0e3e87-47c0-413c-adf2-0f6a439eef01 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:10.475570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0e3e87-47c0-413c-adf2-0f6a439eef01 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:10.475803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0e3e87-47c0-413c-adf2-0f6a439eef01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:10.519803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-7c0e3e87-47c0-413c-adf2-0f6a439eef01 None None] MFA Rules not processed for user `d84b6d05decb47c7b55ac782d5839c1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:10.521720 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-7c0e3e87-47c0-413c-adf2-0f6a439eef01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:10.676782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 834/1666] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:10 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:10.687761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9740b9b4-a400-40f7-823e-6643deac33a5 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:10.688316 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9740b9b4-a400-40f7-823e-6643deac33a5 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:10.689908 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9740b9b4-a400-40f7-823e-6643deac33a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:10.725855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-9740b9b4-a400-40f7-823e-6643deac33a5 None None] MFA Rules not processed for user `d84b6d05decb47c7b55ac782d5839c1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:10.728567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9740b9b4-a400-40f7-823e-6643deac33a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:10.759682 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 833/1667] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:10 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:10.771265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0e3e87-47c0-413c-adf2-0f6a439eef01 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:10.773127 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-7c0e3e87-47c0-413c-adf2-0f6a439eef01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:10.853117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:10.918387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0e3e87-47c0-413c-adf2-0f6a439eef01 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:10.927212 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea527c1-fca7-4a16-adb9-9d364bdea1e9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:10.929451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea527c1-fca7-4a16-adb9-9d364bdea1e9 service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:10.934394 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea527c1-fca7-4a16-adb9-9d364bdea1e9 service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:10.934394 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea527c1-fca7-4a16-adb9-9d364bdea1e9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:10.946683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea527c1-fca7-4a16-adb9-9d364bdea1e9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:10.948362 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea527c1-fca7-4a16-adb9-9d364bdea1e9 service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:10.960697 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 835/1668] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:10 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:11.538240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f251b00e-80db-4038-89c8-39c2d8c2a968 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:11.538736 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f251b00e-80db-4038-89c8-39c2d8c2a968 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:11.539701 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f251b00e-80db-4038-89c8-39c2d8c2a968 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:11.604811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f251b00e-80db-4038-89c8-39c2d8c2a968 None None] MFA Rules not processed for user `bc7137d0ba634b6ab4b83de1ab47c476`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:11.608478 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f251b00e-80db-4038-89c8-39c2d8c2a968 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:11.702088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739f0c3f-913a-46a6-85f7-55c6a6f84e46 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:11.702640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739f0c3f-913a-46a6-85f7-55c6a6f84e46 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:11.702714 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739f0c3f-913a-46a6-85f7-55c6a6f84e46 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:11.704670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 836/1669] 10.222.0.47 () {54 vars in 912 bytes} [Sun Aug 21 01:25:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:25:11.723436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac121908-42b8-4621-8455-1f02249cb350 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:11.723769 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac121908-42b8-4621-8455-1f02249cb350 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:11.724049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac121908-42b8-4621-8455-1f02249cb350 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:11.826498 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 834/1670] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:11 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:25:11.842011 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-ac121908-42b8-4621-8455-1f02249cb350 None None] MFA Rules not processed for user `b532ddab1f5e467989edaa9e4b74bd72`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:11.843770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afcafb4-910a-44ee-86db-18c179c77155 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:11.844215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afcafb4-910a-44ee-86db-18c179c77155 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:11.844498 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afcafb4-910a-44ee-86db-18c179c77155 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:11.849934 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ac121908-42b8-4621-8455-1f02249cb350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:11.901213 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-8afcafb4-910a-44ee-86db-18c179c77155 None None] MFA Rules not processed for user `bc7137d0ba634b6ab4b83de1ab47c476`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:11.902731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8afcafb4-910a-44ee-86db-18c179c77155 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:11.921996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 835/1671] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:11 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:11.941712 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afcafb4-910a-44ee-86db-18c179c77155 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:11.941712 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8afcafb4-910a-44ee-86db-18c179c77155 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:11.966311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afcafb4-910a-44ee-86db-18c179c77155 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:11.990038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3319c24f-b41e-4f3c-80b0-d8d559bfeac9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:11.993510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3319c24f-b41e-4f3c-80b0-d8d559bfeac9 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:11.993510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3319c24f-b41e-4f3c-80b0-d8d559bfeac9 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:11.993510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3319c24f-b41e-4f3c-80b0-d8d559bfeac9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:12.027395 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3319c24f-b41e-4f3c-80b0-d8d559bfeac9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:12.034383 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3319c24f-b41e-4f3c-80b0-d8d559bfeac9 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:12.037381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 837/1672] 10.222.0.47 () {58 vars in 1009 bytes} [Sun Aug 21 01:25:11 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 314 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:12.048636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac121908-42b8-4621-8455-1f02249cb350 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:12.053742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ac121908-42b8-4621-8455-1f02249cb350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:12.061431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 836/1673] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:11 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:12.092335 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac121908-42b8-4621-8455-1f02249cb350 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:12.106271 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27d1319-40c8-46d4-a0f3-b266cda48264 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:12.107453 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27d1319-40c8-46d4-a0f3-b266cda48264 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:12.108078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27d1319-40c8-46d4-a0f3-b266cda48264 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:12.108530 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27d1319-40c8-46d4-a0f3-b266cda48264 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:12.134629 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27d1319-40c8-46d4-a0f3-b266cda48264 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:12.137531 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27d1319-40c8-46d4-a0f3-b266cda48264 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:12.174773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 838/1674] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:25:13.067854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3319c24f-b41e-4f3c-80b0-d8d559bfeac9 service nova] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:13.069658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-3319c24f-b41e-4f3c-80b0-d8d559bfeac9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:13.087821 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3319c24f-b41e-4f3c-80b0-d8d559bfeac9 service nova] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:13.097270 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633e2a77-c3c3-4bc1-bde5-fd7664c15d0f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:13.098045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633e2a77-c3c3-4bc1-bde5-fd7664c15d0f service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:13.098411 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633e2a77-c3c3-4bc1-bde5-fd7664c15d0f service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:13.098911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633e2a77-c3c3-4bc1-bde5-fd7664c15d0f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:13.105565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-633e2a77-c3c3-4bc1-bde5-fd7664c15d0f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:13.109178 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e9507-2184-4b4c-8f69-29984b9ff754 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:13.110016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e9507-2184-4b4c-8f69-29984b9ff754 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:13.110439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e9507-2184-4b4c-8f69-29984b9ff754 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:13.112166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 839/1675] 10.222.0.47 () {54 vars in 912 bytes} [Sun Aug 21 01:25:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:25:13.130236 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a937b175-112b-4d2b-9018-01ebb151c082 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:13.130897 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a937b175-112b-4d2b-9018-01ebb151c082 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:13.131451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a937b175-112b-4d2b-9018-01ebb151c082 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:13.179679 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-a937b175-112b-4d2b-9018-01ebb151c082 None None] MFA Rules not processed for user `b532ddab1f5e467989edaa9e4b74bd72`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:13.181658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-a937b175-112b-4d2b-9018-01ebb151c082 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:13.223384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 840/1676] 10.222.0.47 () {58 vars in 1009 bytes} [Sun Aug 21 01:25:13 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:25:13.247534 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633e2a77-c3c3-4bc1-bde5-fd7664c15d0f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:13.255249 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633e2a77-c3c3-4bc1-bde5-fd7664c15d0f service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:13.324621 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 837/1677] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:13 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:13.325423 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a937b175-112b-4d2b-9018-01ebb151c082 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:13.350132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a937b175-112b-4d2b-9018-01ebb151c082 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:13.371775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f62342-3d06-4d6a-bc1e-0f12ccd2015d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:13.372667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f62342-3d06-4d6a-bc1e-0f12ccd2015d service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:13.373278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f62342-3d06-4d6a-bc1e-0f12ccd2015d service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:13.373835 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f62342-3d06-4d6a-bc1e-0f12ccd2015d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:13.388912 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f62342-3d06-4d6a-bc1e-0f12ccd2015d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:13.390436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f62342-3d06-4d6a-bc1e-0f12ccd2015d service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:13.417283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 841/1678] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:13 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:25:14.010780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b970904d-6976-4f1d-8d42-5ed463ed22d1 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:14.011451 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b970904d-6976-4f1d-8d42-5ed463ed22d1 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:14.011911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b970904d-6976-4f1d-8d42-5ed463ed22d1 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:14.013866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 838/1679] 10.222.0.47 () {54 vars in 961 bytes} [Sun Aug 21 01:25:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:25:14.029748 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5a5f32-d9e3-498e-a603-85c78e24beae None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:14.029748 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5a5f32-d9e3-498e-a603-85c78e24beae None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:14.030174 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5a5f32-d9e3-498e-a603-85c78e24beae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:14.082594 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-3e5a5f32-d9e3-498e-a603-85c78e24beae None None] MFA Rules not processed for user `972b0f1dbfef4d85a6dfc4c5fd839bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:14.091839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3e5a5f32-d9e3-498e-a603-85c78e24beae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:14.315153 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 842/1680] 10.222.0.47 () {58 vars in 1058 bytes} [Sun Aug 21 01:25:14 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:14.329814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b970904d-6976-4f1d-8d42-5ed463ed22d1 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:14.379732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b970904d-6976-4f1d-8d42-5ed463ed22d1 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:14.423615 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421ffa8b-2322-4db4-bb47-dec5e73b4898 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '972b0f1dbfef4d85a6dfc4c5fd839bde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:14.424399 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421ffa8b-2322-4db4-bb47-dec5e73b4898 service placement] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:14.425003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421ffa8b-2322-4db4-bb47-dec5e73b4898 service placement] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:14.425415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421ffa8b-2322-4db4-bb47-dec5e73b4898 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:14.429232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-421ffa8b-2322-4db4-bb47-dec5e73b4898 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:14.472777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421ffa8b-2322-4db4-bb47-dec5e73b4898 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:14.486010 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421ffa8b-2322-4db4-bb47-dec5e73b4898 service placement] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:14.513081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 839/1681] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:14 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:25:15.130101 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5aaf04-ccbf-4ebf-b87a-ae1fde63866a None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:15.130381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5aaf04-ccbf-4ebf-b87a-ae1fde63866a None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:15.130657 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5aaf04-ccbf-4ebf-b87a-ae1fde63866a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:15.181943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:15.200749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-9f5aaf04-ccbf-4ebf-b87a-ae1fde63866a None None] MFA Rules not processed for user `0cd8d822f15943a78575c3744692cdb9`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:15.202060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9f5aaf04-ccbf-4ebf-b87a-ae1fde63866a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:15.248232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e4508d-06c4-46ad-9957-7a139c58ba1e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:15.248517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e4508d-06c4-46ad-9957-7a139c58ba1e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:15.248790 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e4508d-06c4-46ad-9957-7a139c58ba1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:15.317474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-94e4508d-06c4-46ad-9957-7a139c58ba1e None None] MFA Rules not processed for user `d84b6d05decb47c7b55ac782d5839c1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:15.319118 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-94e4508d-06c4-46ad-9957-7a139c58ba1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:15.376713 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 840/1682] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:15 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:15.479268 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 843/1683] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:15 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 350 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:15.950724 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e4508d-06c4-46ad-9957-7a139c58ba1e None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:15.971740 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e4508d-06c4-46ad-9957-7a139c58ba1e None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:15.985720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ae05e1-14f4-49cc-a450-caf43da16929 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:15.986514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ae05e1-14f4-49cc-a450-caf43da16929 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:15.986953 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ae05e1-14f4-49cc-a450-caf43da16929 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:15.987345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ae05e1-14f4-49cc-a450-caf43da16929 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:16.005797 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ae05e1-14f4-49cc-a450-caf43da16929 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:16.007420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ae05e1-14f4-49cc-a450-caf43da16929 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:16.064535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 841/1684] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:15 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:17.251678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5aaf04-ccbf-4ebf-b87a-ae1fde63866a None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:17.371706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5aaf04-ccbf-4ebf-b87a-ae1fde63866a None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:17.385605 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c27c14-c41a-4f14-95fe-38c7d813bad3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:17.390249 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c27c14-c41a-4f14-95fe-38c7d813bad3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:17.390249 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c27c14-c41a-4f14-95fe-38c7d813bad3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:17.390249 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c27c14-c41a-4f14-95fe-38c7d813bad3 admin admin] PATH_INFO: `/v3/users/4017b4ef70b34e4e822005a4ca02dd73` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:17.390249 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c27c14-c41a-4f14-95fe-38c7d813bad3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4017b4ef70b34e4e822005a4ca02dd73)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:17.391155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c27c14-c41a-4f14-95fe-38c7d813bad3 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:17.780233 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b7c27c14-c41a-4f14-95fe-38c7d813bad3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4017b4ef70b34e4e822005a4ca02dd73'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:17.816358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b7c27c14-c41a-4f14-95fe-38c7d813bad3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4017b4ef70b34e4e822005a4ca02dd73'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:17.884669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b7c27c14-c41a-4f14-95fe-38c7d813bad3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4017b4ef70b34e4e822005a4ca02dd73'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:17.972912 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b7c27c14-c41a-4f14-95fe-38c7d813bad3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4017b4ef70b34e4e822005a4ca02dd73'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:17.973570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b7c27c14-c41a-4f14-95fe-38c7d813bad3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4017b4ef70b34e4e822005a4ca02dd73'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:17.987273 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 844/1685] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:17 2022] DELETE /identity/v3/users/4017b4ef70b34e4e822005a4ca02dd73 => generated 0 bytes in 733 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:17.988836 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7edd152-d3a8-42e0-bdf5-46b3c9a81fdb None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:17.989329 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7edd152-d3a8-42e0-bdf5-46b3c9a81fdb None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:17.989582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7edd152-d3a8-42e0-bdf5-46b3c9a81fdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:18.027847 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f7edd152-d3a8-42e0-bdf5-46b3c9a81fdb None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:18.044287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f7edd152-d3a8-42e0-bdf5-46b3c9a81fdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:18.071350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:18.263454 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 842/1686] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 275 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:18.370784 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c27c14-c41a-4f14-95fe-38c7d813bad3 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:18.375849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-b7c27c14-c41a-4f14-95fe-38c7d813bad3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:18.443328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:18.502750 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c27c14-c41a-4f14-95fe-38c7d813bad3 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:18.506481 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3903f46b-6b2b-49a3-b1aa-44ce9a35a462 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:18.507974 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3903f46b-6b2b-49a3-b1aa-44ce9a35a462 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:18.508393 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3903f46b-6b2b-49a3-b1aa-44ce9a35a462 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:18.508879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3903f46b-6b2b-49a3-b1aa-44ce9a35a462 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:18.522668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3903f46b-6b2b-49a3-b1aa-44ce9a35a462 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:18.528970 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3903f46b-6b2b-49a3-b1aa-44ce9a35a462 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:18.534509 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3eb533-efc0-44db-a4d8-46f1395c63ef None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:18.534942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3eb533-efc0-44db-a4d8-46f1395c63ef None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:18.535314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3eb533-efc0-44db-a4d8-46f1395c63ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:18.536783 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 845/1687] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:18.625504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-3e3eb533-efc0-44db-a4d8-46f1395c63ef None None] MFA Rules not processed for user `0f4a35c46e4741c1b238f24d203b416d`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:18.626849 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-3e3eb533-efc0-44db-a4d8-46f1395c63ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:18.869230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 843/1688] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:18 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 340 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:18.892210 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3903f46b-6b2b-49a3-b1aa-44ce9a35a462 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:18.894255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3903f46b-6b2b-49a3-b1aa-44ce9a35a462 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:19.099277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3903f46b-6b2b-49a3-b1aa-44ce9a35a462 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:19.116860 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde0a7d0-2eac-4e3d-8ff6-b03f10ae2d5e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:19.117618 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde0a7d0-2eac-4e3d-8ff6-b03f10ae2d5e service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:19.118042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde0a7d0-2eac-4e3d-8ff6-b03f10ae2d5e service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:19.118375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde0a7d0-2eac-4e3d-8ff6-b03f10ae2d5e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:19.138899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde0a7d0-2eac-4e3d-8ff6-b03f10ae2d5e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:19.139956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde0a7d0-2eac-4e3d-8ff6-b03f10ae2d5e service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:19.171107 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 846/1689] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:18 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 279 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:20.170375 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3eb533-efc0-44db-a4d8-46f1395c63ef None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:20.173651 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-3e3eb533-efc0-44db-a4d8-46f1395c63ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:20.199454 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3eb533-efc0-44db-a4d8-46f1395c63ef None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:20.209444 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c5cc46-f508-4ffb-a699-6ac0a5891bdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:20.211590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c5cc46-f508-4ffb-a699-6ac0a5891bdf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:20.211590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c5cc46-f508-4ffb-a699-6ac0a5891bdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:20.211590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c5cc46-f508-4ffb-a699-6ac0a5891bdf admin admin] PATH_INFO: `/v3/projects/33ee98a5fc5743a59a10cff298eda51f` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:20.216675 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c5cc46-f508-4ffb-a699-6ac0a5891bdf admin admin] RBAC: Authorizing `identity:delete_project(project_id=33ee98a5fc5743a59a10cff298eda51f)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:20.219422 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c5cc46-f508-4ffb-a699-6ac0a5891bdf admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:20.406054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-82c5cc46-f508-4ffb-a699-6ac0a5891bdf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33ee98a5fc5743a59a10cff298eda51f', 'request_id': 'req-82c5cc46-f508-4ffb-a699-6ac0a5891bdf'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:20.413802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-82c5cc46-f508-4ffb-a699-6ac0a5891bdf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33ee98a5fc5743a59a10cff298eda51f', 'request_id': 'req-82c5cc46-f508-4ffb-a699-6ac0a5891bdf'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:20.425218 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-82c5cc46-f508-4ffb-a699-6ac0a5891bdf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33ee98a5fc5743a59a10cff298eda51f', 'request_id': 'req-82c5cc46-f508-4ffb-a699-6ac0a5891bdf'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:20.501997 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-82c5cc46-f508-4ffb-a699-6ac0a5891bdf admin admin] The token cache is being invalidate because project 33ee98a5fc5743a59a10cff298eda51f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:20.502290 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-82c5cc46-f508-4ffb-a699-6ac0a5891bdf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:20.504233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-82c5cc46-f508-4ffb-a699-6ac0a5891bdf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:20.507031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 844/1690] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:20 2022] DELETE /identity/v3/projects/33ee98a5fc5743a59a10cff298eda51f => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:20.511767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde0a7d0-2eac-4e3d-8ff6-b03f10ae2d5e service nova] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:20.535536 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde0a7d0-2eac-4e3d-8ff6-b03f10ae2d5e service nova] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:20.656291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78cb5a0-8c9d-4f85-9b27-efbbb36d4900 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:20.657058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78cb5a0-8c9d-4f85-9b27-efbbb36d4900 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:20.657248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78cb5a0-8c9d-4f85-9b27-efbbb36d4900 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:20.657447 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78cb5a0-8c9d-4f85-9b27-efbbb36d4900 admin admin] PATH_INFO: `/v3/users/6e4f494b3ea04cd7bb9a0ab22a890259` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:20.659408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78cb5a0-8c9d-4f85-9b27-efbbb36d4900 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e4f494b3ea04cd7bb9a0ab22a890259)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:20.661635 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78cb5a0-8c9d-4f85-9b27-efbbb36d4900 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:20.941099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b78cb5a0-8c9d-4f85-9b27-efbbb36d4900 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e4f494b3ea04cd7bb9a0ab22a890259'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:20.947832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:20.961119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b78cb5a0-8c9d-4f85-9b27-efbbb36d4900 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e4f494b3ea04cd7bb9a0ab22a890259'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:20.981501 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b78cb5a0-8c9d-4f85-9b27-efbbb36d4900 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e4f494b3ea04cd7bb9a0ab22a890259'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:20.983204 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced9e697-d68b-4863-80dd-79165542148d None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:20.983603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced9e697-d68b-4863-80dd-79165542148d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:20.983978 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced9e697-d68b-4863-80dd-79165542148d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:21.033235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-ced9e697-d68b-4863-80dd-79165542148d None None] MFA Rules not processed for user `0cd8d822f15943a78575c3744692cdb9`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:21.036433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ced9e697-d68b-4863-80dd-79165542148d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:21.080602 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:21.085769 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b78cb5a0-8c9d-4f85-9b27-efbbb36d4900 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e4f494b3ea04cd7bb9a0ab22a890259'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:21.086432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b78cb5a0-8c9d-4f85-9b27-efbbb36d4900 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e4f494b3ea04cd7bb9a0ab22a890259'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:21.099170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 847/1691] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:20 2022] DELETE /identity/v3/users/6e4f494b3ea04cd7bb9a0ab22a890259 => generated 0 bytes in 585 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:21.157305 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 845/1692] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:20 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:21.713063 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78cb5a0-8c9d-4f85-9b27-efbbb36d4900 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:21.714650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-b78cb5a0-8c9d-4f85-9b27-efbbb36d4900 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:21.825947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78cb5a0-8c9d-4f85-9b27-efbbb36d4900 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:21.825947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be88d60-6954-4c67-aa7f-a8ffddc2a554 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:21.825947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be88d60-6954-4c67-aa7f-a8ffddc2a554 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:21.825947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be88d60-6954-4c67-aa7f-a8ffddc2a554 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:21.826480 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be88d60-6954-4c67-aa7f-a8ffddc2a554 admin admin] PATH_INFO: `/v3/projects/7a76b10f0fd0405c93444e074e0db38b` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:21.826927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be88d60-6954-4c67-aa7f-a8ffddc2a554 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a76b10f0fd0405c93444e074e0db38b)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:21.829018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be88d60-6954-4c67-aa7f-a8ffddc2a554 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:22.007530 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-1be88d60-6954-4c67-aa7f-a8ffddc2a554 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a76b10f0fd0405c93444e074e0db38b', 'request_id': 'req-1be88d60-6954-4c67-aa7f-a8ffddc2a554'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:22.081053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-1be88d60-6954-4c67-aa7f-a8ffddc2a554 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a76b10f0fd0405c93444e074e0db38b', 'request_id': 'req-1be88d60-6954-4c67-aa7f-a8ffddc2a554'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:22.108195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-1be88d60-6954-4c67-aa7f-a8ffddc2a554 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a76b10f0fd0405c93444e074e0db38b', 'request_id': 'req-1be88d60-6954-4c67-aa7f-a8ffddc2a554'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:22.134420 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-1be88d60-6954-4c67-aa7f-a8ffddc2a554 admin admin] The token cache is being invalidate because project 7a76b10f0fd0405c93444e074e0db38b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:22.135023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-1be88d60-6954-4c67-aa7f-a8ffddc2a554 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:22.135929 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-1be88d60-6954-4c67-aa7f-a8ffddc2a554 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:22.143587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f597c6f9-4e5e-4113-9f9d-64b7ed56313a None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:22.144734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f597c6f9-4e5e-4113-9f9d-64b7ed56313a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:22.145223 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f597c6f9-4e5e-4113-9f9d-64b7ed56313a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:22.148708 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 848/1693] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:21 2022] DELETE /identity/v3/projects/7a76b10f0fd0405c93444e074e0db38b => generated 0 bytes in 425 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:22.177019 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f597c6f9-4e5e-4113-9f9d-64b7ed56313a None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:22.179433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f597c6f9-4e5e-4113-9f9d-64b7ed56313a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:22.322151 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 846/1694] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:22.336321 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d3184-425d-4013-924d-4279dae1b887 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:22.336607 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d3184-425d-4013-924d-4279dae1b887 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:22.337132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d3184-425d-4013-924d-4279dae1b887 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:22.365564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-ac8d3184-425d-4013-924d-4279dae1b887 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:22.366642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ac8d3184-425d-4013-924d-4279dae1b887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:22.380416 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 849/1695] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:22.381704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f597c6f9-4e5e-4113-9f9d-64b7ed56313a None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:22.417648 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f597c6f9-4e5e-4113-9f9d-64b7ed56313a None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:22.425727 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6683907e-c976-4a1e-9664-fe6046bad1c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:22.426995 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6683907e-c976-4a1e-9664-fe6046bad1c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:22.427278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6683907e-c976-4a1e-9664-fe6046bad1c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:22.427550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6683907e-c976-4a1e-9664-fe6046bad1c5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:22.428265 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6683907e-c976-4a1e-9664-fe6046bad1c5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:25:22.428575 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6683907e-c976-4a1e-9664-fe6046bad1c5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:22.431307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6683907e-c976-4a1e-9664-fe6046bad1c5 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:22.450001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 847/1696] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:25:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:25:22.454112 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8d3184-425d-4013-924d-4279dae1b887 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:22.467936 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8d3184-425d-4013-924d-4279dae1b887 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:22.475392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e213c6c-2064-4923-9d48-e67668d08fa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:22.477122 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e213c6c-2064-4923-9d48-e67668d08fa9 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:22.477489 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e213c6c-2064-4923-9d48-e67668d08fa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:22.477840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e213c6c-2064-4923-9d48-e67668d08fa9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:22.478711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e213c6c-2064-4923-9d48-e67668d08fa9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:22.481750 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e213c6c-2064-4923-9d48-e67668d08fa9 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:22.603686 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 850/1697] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:22 2022] POST /identity/v3/projects => generated 369 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:22.606742 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6683907e-c976-4a1e-9664-fe6046bad1c5 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:22.622043 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6683907e-c976-4a1e-9664-fe6046bad1c5 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:22.717462 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbeb361-d6a3-47b1-8695-fe74aca08cd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:22.718001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbeb361-d6a3-47b1-8695-fe74aca08cd3 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:22.718334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbeb361-d6a3-47b1-8695-fe74aca08cd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:22.718749 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbeb361-d6a3-47b1-8695-fe74aca08cd3 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:22.719780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbeb361-d6a3-47b1-8695-fe74aca08cd3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:22.722652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbeb361-d6a3-47b1-8695-fe74aca08cd3 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:22.825108 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 848/1698] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:22 2022] POST /identity/v3/users => generated 285 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:25:22.826442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e213c6c-2064-4923-9d48-e67668d08fa9 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:22.841789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e213c6c-2064-4923-9d48-e67668d08fa9 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:22.848630 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d248185-19bd-44aa-a2d5-331cb4474e49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:22.849987 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d248185-19bd-44aa-a2d5-331cb4474e49 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:22.850422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d248185-19bd-44aa-a2d5-331cb4474e49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:22.850868 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d248185-19bd-44aa-a2d5-331cb4474e49 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:22.851705 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d248185-19bd-44aa-a2d5-331cb4474e49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:22.854192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d248185-19bd-44aa-a2d5-331cb4474e49 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:22.887301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 851/1699] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:22 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:22.890371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbeb361-d6a3-47b1-8695-fe74aca08cd3 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:22.906497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbeb361-d6a3-47b1-8695-fe74aca08cd3 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:22.912016 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7d55b7-9828-43ba-81a5-410fe37fa5a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:22.912783 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7d55b7-9828-43ba-81a5-410fe37fa5a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:22.912937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7d55b7-9828-43ba-81a5-410fe37fa5a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:22.913234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7d55b7-9828-43ba-81a5-410fe37fa5a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:22.913941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7d55b7-9828-43ba-81a5-410fe37fa5a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:22.916554 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7d55b7-9828-43ba-81a5-410fe37fa5a0 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:22.946614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 849/1700] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:22 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:22.946912 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d248185-19bd-44aa-a2d5-331cb4474e49 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:22.962465 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d248185-19bd-44aa-a2d5-331cb4474e49 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:22.968224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df57825-69ba-4c50-b318-ad0955113541 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:22.974218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df57825-69ba-4c50-b318-ad0955113541 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:22.974218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df57825-69ba-4c50-b318-ad0955113541 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:22.974218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df57825-69ba-4c50-b318-ad0955113541 admin admin] PATH_INFO: `/v3/projects/9700cc0520bb44f59dcede462a56bcbc/users/2787f295dc064a9bb96b70bd651df3d8/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:22.994799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df57825-69ba-4c50-b318-ad0955113541 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9700cc0520bb44f59dcede462a56bcbc, user_id=2787f295dc064a9bb96b70bd651df3d8, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:22.998857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df57825-69ba-4c50-b318-ad0955113541 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:23.001880 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=87,keystone:DELETE=10,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:23.093614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 852/1701] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:22 2022] PUT /identity/v3/projects/9700cc0520bb44f59dcede462a56bcbc/users/2787f295dc064a9bb96b70bd651df3d8/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:25:23.107550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f028de-6974-480c-af90-85f8bac3bb77 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:23.107550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f028de-6974-480c-af90-85f8bac3bb77 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:23.107550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f028de-6974-480c-af90-85f8bac3bb77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:23.189385 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-d9f028de-6974-480c-af90-85f8bac3bb77 None None] MFA Rules not processed for user `2787f295dc064a9bb96b70bd651df3d8`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:23.191436 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d9f028de-6974-480c-af90-85f8bac3bb77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:23.225009 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:23.315776 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 850/1702] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:23 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:23.321269 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df57825-69ba-4c50-b318-ad0955113541 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:23.540627 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df57825-69ba-4c50-b318-ad0955113541 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:23.559787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8e26e3-c67e-4201-bb44-274a3d793d4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:23.560449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8e26e3-c67e-4201-bb44-274a3d793d4e admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:23.567873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8e26e3-c67e-4201-bb44-274a3d793d4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:23.567873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8e26e3-c67e-4201-bb44-274a3d793d4e admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:23.567873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8e26e3-c67e-4201-bb44-274a3d793d4e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:23.567873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8e26e3-c67e-4201-bb44-274a3d793d4e admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:23.789807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f028de-6974-480c-af90-85f8bac3bb77 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:23.793464 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 853/1703] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:23 2022] POST /identity/v3/projects => generated 369 bytes in 463 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:23.821654 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f028de-6974-480c-af90-85f8bac3bb77 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:23.909868 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e0197a-49de-4e5e-b71b-ced6bc7b29bf None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:23.922127 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e0197a-49de-4e5e-b71b-ced6bc7b29bf None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:23.922127 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e0197a-49de-4e5e-b71b-ced6bc7b29bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:23.957853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-36e0197a-49de-4e5e-b71b-ced6bc7b29bf None None] MFA Rules not processed for user `0f4a35c46e4741c1b238f24d203b416d`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:23.960247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-36e0197a-49de-4e5e-b71b-ced6bc7b29bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:24.066495 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da48494-12ee-42d0-a511-a5bc5cecf447 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:24.066684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da48494-12ee-42d0-a511-a5bc5cecf447 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:24.067215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da48494-12ee-42d0-a511-a5bc5cecf447 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:24.067468 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da48494-12ee-42d0-a511-a5bc5cecf447 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:24.068587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da48494-12ee-42d0-a511-a5bc5cecf447 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:24.071623 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da48494-12ee-42d0-a511-a5bc5cecf447 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:24.213830 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 854/1704] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:23 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:25:24.220434 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 851/1705] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:23 2022] POST /identity/v3/users => generated 291 bytes in 433 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:24.226103 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e0197a-49de-4e5e-b71b-ced6bc7b29bf None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:24.239976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e0197a-49de-4e5e-b71b-ced6bc7b29bf None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:24.246150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0575802f-5fae-4c22-8fab-c8541bbe0c1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:24.247044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0575802f-5fae-4c22-8fab-c8541bbe0c1e admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:24.247418 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0575802f-5fae-4c22-8fab-c8541bbe0c1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:24.260641 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0575802f-5fae-4c22-8fab-c8541bbe0c1e admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:24.261641 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0575802f-5fae-4c22-8fab-c8541bbe0c1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:24.280673 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0575802f-5fae-4c22-8fab-c8541bbe0c1e admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:24.330486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 855/1706] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:24 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:24.331035 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da48494-12ee-42d0-a511-a5bc5cecf447 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:24.364865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da48494-12ee-42d0-a511-a5bc5cecf447 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:24.382103 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96568df7-a786-4ac0-ab70-bd8bdfbb5af4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:24.393492 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96568df7-a786-4ac0-ab70-bd8bdfbb5af4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:24.393492 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96568df7-a786-4ac0-ab70-bd8bdfbb5af4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:24.393492 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96568df7-a786-4ac0-ab70-bd8bdfbb5af4 admin admin] PATH_INFO: `/v3/projects/e3d46a17d2444744a6200cbca7471fc3/users/3852478ed0ce4b49b1f7677a962b9453/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:24.410068 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96568df7-a786-4ac0-ab70-bd8bdfbb5af4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e3d46a17d2444744a6200cbca7471fc3, user_id=3852478ed0ce4b49b1f7677a962b9453, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:24.423887 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c2f73f-6be1-47a9-9fcb-5c76b2707ebf None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:24.424175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c2f73f-6be1-47a9-9fcb-5c76b2707ebf None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:24.424574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c2f73f-6be1-47a9-9fcb-5c76b2707ebf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:24.425278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96568df7-a786-4ac0-ab70-bd8bdfbb5af4 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:24.530809 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 852/1707] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:24 2022] PUT /identity/v3/projects/e3d46a17d2444744a6200cbca7471fc3/users/3852478ed0ce4b49b1f7677a962b9453/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:24.530809 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96568df7-a786-4ac0-ab70-bd8bdfbb5af4 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:24.536071 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-96568df7-a786-4ac0-ab70-bd8bdfbb5af4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:24.541147 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-71c2f73f-6be1-47a9-9fcb-5c76b2707ebf None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:24.542610 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-71c2f73f-6be1-47a9-9fcb-5c76b2707ebf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:24.667010 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96568df7-a786-4ac0-ab70-bd8bdfbb5af4 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:24.673983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 856/1708] 10.222.0.47 () {58 vars in 1009 bytes} [Sun Aug 21 01:25:24 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:25:24.675799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c2f73f-6be1-47a9-9fcb-5c76b2707ebf None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:24.685089 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1ead6d-8f96-4934-aa71-82cb68854d7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:24.685089 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1ead6d-8f96-4934-aa71-82cb68854d7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:24.685605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1ead6d-8f96-4934-aa71-82cb68854d7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:24.685605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1ead6d-8f96-4934-aa71-82cb68854d7f admin admin] PATH_INFO: `/v3/users/dc3e6a6484c74d2cab56e6696404817d` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:24.688228 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1ead6d-8f96-4934-aa71-82cb68854d7f admin admin] RBAC: Authorizing `identity:delete_user(user_id=dc3e6a6484c74d2cab56e6696404817d)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:24.692094 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1ead6d-8f96-4934-aa71-82cb68854d7f admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:24.700141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c2f73f-6be1-47a9-9fcb-5c76b2707ebf None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:24.707500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d76f24-4b43-454e-9f66-4679fb844621 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:24.713572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d76f24-4b43-454e-9f66-4679fb844621 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:24.713572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d76f24-4b43-454e-9f66-4679fb844621 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:24.713572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d76f24-4b43-454e-9f66-4679fb844621 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:24.713572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d76f24-4b43-454e-9f66-4679fb844621 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:24.713572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d76f24-4b43-454e-9f66-4679fb844621 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:24.737810 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 857/1709] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:24 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:24.742688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d76f24-4b43-454e-9f66-4679fb844621 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:24.761089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d76f24-4b43-454e-9f66-4679fb844621 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:24.767599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3872ed31-bb8c-4646-b13a-c453213b7a0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:24.768538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3872ed31-bb8c-4646-b13a-c453213b7a0a admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:24.768961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3872ed31-bb8c-4646-b13a-c453213b7a0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:24.769360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3872ed31-bb8c-4646-b13a-c453213b7a0a admin admin] PATH_INFO: `/v3/projects/e3d46a17d2444744a6200cbca7471fc3/users/3852478ed0ce4b49b1f7677a962b9453/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:24.771676 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3872ed31-bb8c-4646-b13a-c453213b7a0a admin admin] RBAC: Authorizing `identity:create_grant(project_id=e3d46a17d2444744a6200cbca7471fc3, user_id=3852478ed0ce4b49b1f7677a962b9453, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:24.775073 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3872ed31-bb8c-4646-b13a-c453213b7a0a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:24.791589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 858/1710] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:24 2022] PUT /identity/v3/projects/e3d46a17d2444744a6200cbca7471fc3/users/3852478ed0ce4b49b1f7677a962b9453/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:24.801795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0b7880-bc93-426d-82e9-d8252c0bb234 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:24.802054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0b7880-bc93-426d-82e9-d8252c0bb234 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:24.802384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0b7880-bc93-426d-82e9-d8252c0bb234 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:24.840735 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-7a0b7880-bc93-426d-82e9-d8252c0bb234 None None] MFA Rules not processed for user `3852478ed0ce4b49b1f7677a962b9453`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:24.851625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-7a0b7880-bc93-426d-82e9-d8252c0bb234 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:24.863520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:24.917657 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=91,keystone:INSERT=4,keystone:DELETE=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:24.939047 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7b1ead6d-8f96-4934-aa71-82cb68854d7f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dc3e6a6484c74d2cab56e6696404817d'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:24.949573 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7b1ead6d-8f96-4934-aa71-82cb68854d7f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dc3e6a6484c74d2cab56e6696404817d'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:24.951518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7b1ead6d-8f96-4934-aa71-82cb68854d7f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dc3e6a6484c74d2cab56e6696404817d'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:24.969748 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7b1ead6d-8f96-4934-aa71-82cb68854d7f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dc3e6a6484c74d2cab56e6696404817d'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:24.984193 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 859/1711] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:24 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:25:24.996292 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0b7880-bc93-426d-82e9-d8252c0bb234 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:24.997798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-7a0b7880-bc93-426d-82e9-d8252c0bb234 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:25.067700 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7b1ead6d-8f96-4934-aa71-82cb68854d7f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dc3e6a6484c74d2cab56e6696404817d'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:25.071169 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0b7880-bc93-426d-82e9-d8252c0bb234 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:25.076636 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 853/1712] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:24 2022] DELETE /identity/v3/users/dc3e6a6484c74d2cab56e6696404817d => generated 0 bytes in 547 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:25.077601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1ead6d-8f96-4934-aa71-82cb68854d7f admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:25.081251 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7b1ead6d-8f96-4934-aa71-82cb68854d7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:25.109528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af6e5e5-757c-470b-91ef-9ad6e13fd38f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:25.109528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af6e5e5-757c-470b-91ef-9ad6e13fd38f service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:25.109528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af6e5e5-757c-470b-91ef-9ad6e13fd38f service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:25.109528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af6e5e5-757c-470b-91ef-9ad6e13fd38f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:25.117432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-4af6e5e5-757c-470b-91ef-9ad6e13fd38f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:25.141025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af6e5e5-757c-470b-91ef-9ad6e13fd38f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:25.146171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af6e5e5-757c-470b-91ef-9ad6e13fd38f service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:25.170663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 860/1713] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:24 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:25:25.174744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b40af54-b370-4b90-b6cd-b6e8519e7fac None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:25.175197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b40af54-b370-4b90-b6cd-b6e8519e7fac None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:25.175578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b40af54-b370-4b90-b6cd-b6e8519e7fac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:25.233093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-6b40af54-b370-4b90-b6cd-b6e8519e7fac None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:25.234495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6b40af54-b370-4b90-b6cd-b6e8519e7fac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:25.288680 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1ead6d-8f96-4934-aa71-82cb68854d7f admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:25.298914 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22481ada-6641-4da3-b9f6-3fc57c20fdf5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:25.300312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22481ada-6641-4da3-b9f6-3fc57c20fdf5 service swift] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:25.301539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22481ada-6641-4da3-b9f6-3fc57c20fdf5 service swift] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:25.301918 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22481ada-6641-4da3-b9f6-3fc57c20fdf5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:25.311266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-22481ada-6641-4da3-b9f6-3fc57c20fdf5 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:25.386179 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 861/1714] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:25.387369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b40af54-b370-4b90-b6cd-b6e8519e7fac None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:25.389676 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6b40af54-b370-4b90-b6cd-b6e8519e7fac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:25.449104 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22481ada-6641-4da3-b9f6-3fc57c20fdf5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:25.450213 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22481ada-6641-4da3-b9f6-3fc57c20fdf5 service swift] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:25.472221 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 854/1715] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:25:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 21 01:25:25.477261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22481ada-6641-4da3-b9f6-3fc57c20fdf5 service swift] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:25.489403 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-22481ada-6641-4da3-b9f6-3fc57c20fdf5 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:25.500389 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b40af54-b370-4b90-b6cd-b6e8519e7fac None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:25.537092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4484cb2c-8e1c-4311-ad25-e13ca3d423e3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:25.543156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4484cb2c-8e1c-4311-ad25-e13ca3d423e3 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:25.543156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4484cb2c-8e1c-4311-ad25-e13ca3d423e3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:25.543156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4484cb2c-8e1c-4311-ad25-e13ca3d423e3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:25.553265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-4484cb2c-8e1c-4311-ad25-e13ca3d423e3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:25.563413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22481ada-6641-4da3-b9f6-3fc57c20fdf5 service swift] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:25.568989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8e7c3d-4c03-4284-9dee-054cd67182e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:25.570349 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8e7c3d-4c03-4284-9dee-054cd67182e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:25.570679 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8e7c3d-4c03-4284-9dee-054cd67182e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:25.570990 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8e7c3d-4c03-4284-9dee-054cd67182e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:25.582995 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8e7c3d-4c03-4284-9dee-054cd67182e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:25.584593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8e7c3d-4c03-4284-9dee-054cd67182e9 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:25.607186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 855/1716] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:25.766564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4484cb2c-8e1c-4311-ad25-e13ca3d423e3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:25.766564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4484cb2c-8e1c-4311-ad25-e13ca3d423e3 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:25.766564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 862/1717] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:25 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:25.885425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:26.535886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:26.849127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8e7c3d-4c03-4284-9dee-054cd67182e9 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:26.850630 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8c8e7c3d-4c03-4284-9dee-054cd67182e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:26.872014 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8e7c3d-4c03-4284-9dee-054cd67182e9 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:26.879150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a614c5-eacc-4a04-a191-f6e7b797e188 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:26.880094 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a614c5-eacc-4a04-a191-f6e7b797e188 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:26.880768 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a614c5-eacc-4a04-a191-f6e7b797e188 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:26.881187 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a614c5-eacc-4a04-a191-f6e7b797e188 admin admin] PATH_INFO: `/v3/projects/1f324851bf32439880d91a81160858ab` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:26.885846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a614c5-eacc-4a04-a191-f6e7b797e188 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1f324851bf32439880d91a81160858ab)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:26.886222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a614c5-eacc-4a04-a191-f6e7b797e188 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:27.148296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c5a614c5-eacc-4a04-a191-f6e7b797e188 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1f324851bf32439880d91a81160858ab', 'request_id': 'req-c5a614c5-eacc-4a04-a191-f6e7b797e188'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:27.154175 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c5a614c5-eacc-4a04-a191-f6e7b797e188 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1f324851bf32439880d91a81160858ab', 'request_id': 'req-c5a614c5-eacc-4a04-a191-f6e7b797e188'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:27.169083 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c5a614c5-eacc-4a04-a191-f6e7b797e188 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1f324851bf32439880d91a81160858ab', 'request_id': 'req-c5a614c5-eacc-4a04-a191-f6e7b797e188'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:27.310228 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c5a614c5-eacc-4a04-a191-f6e7b797e188 admin admin] The token cache is being invalidate because project 1f324851bf32439880d91a81160858ab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:27.310531 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c5a614c5-eacc-4a04-a191-f6e7b797e188 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:27.311213 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c5a614c5-eacc-4a04-a191-f6e7b797e188 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:27.323496 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 856/1718] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:26 2022] DELETE /identity/v3/projects/1f324851bf32439880d91a81160858ab => generated 0 bytes in 471 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:27.325915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2a1589-df4f-4a0e-aae0-90918cb6ba61 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:27.327704 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2a1589-df4f-4a0e-aae0-90918cb6ba61 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:27.328257 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2a1589-df4f-4a0e-aae0-90918cb6ba61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:27.382014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-de2a1589-df4f-4a0e-aae0-90918cb6ba61 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:27.383893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-de2a1589-df4f-4a0e-aae0-90918cb6ba61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:27.538098 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 863/1719] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:25:27.546569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a567b13e-a988-47ec-8108-b717f08f9f68 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:27.547224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a567b13e-a988-47ec-8108-b717f08f9f68 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:27.547224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a567b13e-a988-47ec-8108-b717f08f9f68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:27.583894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-a567b13e-a988-47ec-8108-b717f08f9f68 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:27.589822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a567b13e-a988-47ec-8108-b717f08f9f68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:27.630244 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 857/1720] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:27.635540 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2a1589-df4f-4a0e-aae0-90918cb6ba61 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:27.662119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2a1589-df4f-4a0e-aae0-90918cb6ba61 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:27.690002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e6c73b-cde4-4d1c-8cae-9cdf9e65f0f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:27.690002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e6c73b-cde4-4d1c-8cae-9cdf9e65f0f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:27.690002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e6c73b-cde4-4d1c-8cae-9cdf9e65f0f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:27.690002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e6c73b-cde4-4d1c-8cae-9cdf9e65f0f7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:27.690985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e6c73b-cde4-4d1c-8cae-9cdf9e65f0f7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:25:27.690985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e6c73b-cde4-4d1c-8cae-9cdf9e65f0f7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:27.690985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e6c73b-cde4-4d1c-8cae-9cdf9e65f0f7 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:27.721367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 864/1721] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:25:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:25:27.725735 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a567b13e-a988-47ec-8108-b717f08f9f68 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:27.740047 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a567b13e-a988-47ec-8108-b717f08f9f68 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:27.746705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f81736-4cb9-498f-b4ea-420f9b12f600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:27.748939 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f81736-4cb9-498f-b4ea-420f9b12f600 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:27.749592 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f81736-4cb9-498f-b4ea-420f9b12f600 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:27.750161 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f81736-4cb9-498f-b4ea-420f9b12f600 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:27.750993 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f81736-4cb9-498f-b4ea-420f9b12f600 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:27.753460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f81736-4cb9-498f-b4ea-420f9b12f600 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:27.795880 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ef9be7-88b7-4e91-8e2a-35b44ab628bb None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:27.796222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ef9be7-88b7-4e91-8e2a-35b44ab628bb None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:27.796477 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ef9be7-88b7-4e91-8e2a-35b44ab628bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:27.854457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-47ef9be7-88b7-4e91-8e2a-35b44ab628bb None None] MFA Rules not processed for user `b532ddab1f5e467989edaa9e4b74bd72`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:27.858884 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-47ef9be7-88b7-4e91-8e2a-35b44ab628bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:27.943561 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 858/1722] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:27 2022] POST /identity/v3/projects => generated 373 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:27.945703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f81736-4cb9-498f-b4ea-420f9b12f600 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:27.985982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:28.041465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f81736-4cb9-498f-b4ea-420f9b12f600 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:28.059154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bab6b3-ce0a-4830-9ba5-584cc77857b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:28.072864 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bab6b3-ce0a-4830-9ba5-584cc77857b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:28.073590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bab6b3-ce0a-4830-9ba5-584cc77857b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:28.074107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bab6b3-ce0a-4830-9ba5-584cc77857b4 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:28.075200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bab6b3-ce0a-4830-9ba5-584cc77857b4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:28.078577 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bab6b3-ce0a-4830-9ba5-584cc77857b4 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:28.145355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 865/1723] 10.222.0.67 () {58 vars in 1015 bytes} [Sun Aug 21 01:25:27 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 350 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:28.244981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ef9be7-88b7-4e91-8e2a-35b44ab628bb None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:28.244981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-47ef9be7-88b7-4e91-8e2a-35b44ab628bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:28.244981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ef9be7-88b7-4e91-8e2a-35b44ab628bb None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:28.315189 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 859/1724] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:27 2022] POST /identity/v3/users => generated 287 bytes in 370 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:28.316386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bab6b3-ce0a-4830-9ba5-584cc77857b4 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:28.326097 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-71bab6b3-ce0a-4830-9ba5-584cc77857b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:28.447056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcc3f16-d6d1-49e9-b4f9-ac6b0f037dbb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:28.462960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcc3f16-d6d1-49e9-b4f9-ac6b0f037dbb service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:28.464196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcc3f16-d6d1-49e9-b4f9-ac6b0f037dbb service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:28.464493 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcc3f16-d6d1-49e9-b4f9-ac6b0f037dbb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:28.505528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcc3f16-d6d1-49e9-b4f9-ac6b0f037dbb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:28.513709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcc3f16-d6d1-49e9-b4f9-ac6b0f037dbb service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:28.519315 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bab6b3-ce0a-4830-9ba5-584cc77857b4 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:28.536140 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 866/1725] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:28 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:25:28.549793 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6171d715-b9ce-4c95-a3b8-b33aa7b3c485 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:28.550585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcc3f16-d6d1-49e9-b4f9-ac6b0f037dbb service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:28.551717 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6171d715-b9ce-4c95-a3b8-b33aa7b3c485 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:28.552005 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6171d715-b9ce-4c95-a3b8-b33aa7b3c485 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:28.552311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6171d715-b9ce-4c95-a3b8-b33aa7b3c485 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:28.572346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6171d715-b9ce-4c95-a3b8-b33aa7b3c485 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:28.577734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6171d715-b9ce-4c95-a3b8-b33aa7b3c485 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:28.614386 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcc3f16-d6d1-49e9-b4f9-ac6b0f037dbb service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:28.620246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 860/1726] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:28 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:28.625696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6171d715-b9ce-4c95-a3b8-b33aa7b3c485 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:28.658197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f34ce4-749a-4604-adb7-317dc8c50e0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:28.661218 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6171d715-b9ce-4c95-a3b8-b33aa7b3c485 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:28.666776 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f34ce4-749a-4604-adb7-317dc8c50e0b service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:28.667099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f34ce4-749a-4604-adb7-317dc8c50e0b service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:28.667380 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f34ce4-749a-4604-adb7-317dc8c50e0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:28.679397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5873fd89-4c69-4ff1-b402-dc4622f8e949 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:28.681290 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5873fd89-4c69-4ff1-b402-dc4622f8e949 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:28.681761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5873fd89-4c69-4ff1-b402-dc4622f8e949 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:28.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5873fd89-4c69-4ff1-b402-dc4622f8e949 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:28.685246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5873fd89-4c69-4ff1-b402-dc4622f8e949 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:28.687126 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5873fd89-4c69-4ff1-b402-dc4622f8e949 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:28.703193 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f34ce4-749a-4604-adb7-317dc8c50e0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:28.705288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 861/1727] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:28 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:28.706979 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5873fd89-4c69-4ff1-b402-dc4622f8e949 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:28.709342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f34ce4-749a-4604-adb7-317dc8c50e0b service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:28.721768 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5873fd89-4c69-4ff1-b402-dc4622f8e949 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:28.723460 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 867/1728] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:28 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:28.728496 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a136aa68-6708-4725-80e2-d9ff9ac1c357 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:28.729580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a136aa68-6708-4725-80e2-d9ff9ac1c357 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:28.730008 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a136aa68-6708-4725-80e2-d9ff9ac1c357 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:28.730438 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a136aa68-6708-4725-80e2-d9ff9ac1c357 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:28.731229 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a136aa68-6708-4725-80e2-d9ff9ac1c357 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:28.754137 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a136aa68-6708-4725-80e2-d9ff9ac1c357 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:28.771351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 862/1729] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:28 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:28.774781 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f34ce4-749a-4604-adb7-317dc8c50e0b service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:28.800803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f34ce4-749a-4604-adb7-317dc8c50e0b service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:28.822929 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a578019c-3cd4-4c2e-9cdc-b87478a869c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:28.824997 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a578019c-3cd4-4c2e-9cdc-b87478a869c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:28.826197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a578019c-3cd4-4c2e-9cdc-b87478a869c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:28.827365 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a578019c-3cd4-4c2e-9cdc-b87478a869c4 admin admin] PATH_INFO: `/v3/projects/b009d4c2e5f64d8b8b6b5c702067f8bf/users/c202efb38590421b970f80f9d693122f/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:28.894166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a578019c-3cd4-4c2e-9cdc-b87478a869c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b009d4c2e5f64d8b8b6b5c702067f8bf, user_id=c202efb38590421b970f80f9d693122f, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:28.898587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a578019c-3cd4-4c2e-9cdc-b87478a869c4 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:29.066022 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 868/1730] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:28 2022] PUT /identity/v3/projects/b009d4c2e5f64d8b8b6b5c702067f8bf/users/c202efb38590421b970f80f9d693122f/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:29.066146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70b0bd0-0fd9-4891-871a-69ed6d68eb6d None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:29.066146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70b0bd0-0fd9-4891-871a-69ed6d68eb6d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:29.066146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70b0bd0-0fd9-4891-871a-69ed6d68eb6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:29.141753 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-b70b0bd0-0fd9-4891-871a-69ed6d68eb6d None None] MFA Rules not processed for user `c202efb38590421b970f80f9d693122f`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:29.141753 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b70b0bd0-0fd9-4891-871a-69ed6d68eb6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:29.397467 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 863/1731] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:29 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 353 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:29.399700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a578019c-3cd4-4c2e-9cdc-b87478a869c4 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:29.489261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:29.570643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a578019c-3cd4-4c2e-9cdc-b87478a869c4 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:29.576776 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef94867-0190-4ab0-bf39-8cbdcd0ce163 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:29.577496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef94867-0190-4ab0-bf39-8cbdcd0ce163 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:29.578072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef94867-0190-4ab0-bf39-8cbdcd0ce163 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:29.578423 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef94867-0190-4ab0-bf39-8cbdcd0ce163 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:29.579222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef94867-0190-4ab0-bf39-8cbdcd0ce163 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:29.581794 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef94867-0190-4ab0-bf39-8cbdcd0ce163 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:29.755492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 869/1732] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:29 2022] POST /identity/v3/projects => generated 373 bytes in 358 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:29.761142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70b0bd0-0fd9-4891-871a-69ed6d68eb6d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:29.800029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70b0bd0-0fd9-4891-871a-69ed6d68eb6d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:29.958684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d520e4d7-e3b0-4624-bc25-35924dc6c8d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:29.969088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d520e4d7-e3b0-4624-bc25-35924dc6c8d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:29.969088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d520e4d7-e3b0-4624-bc25-35924dc6c8d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:29.969088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d520e4d7-e3b0-4624-bc25-35924dc6c8d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:29.969088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d520e4d7-e3b0-4624-bc25-35924dc6c8d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:29.970982 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d520e4d7-e3b0-4624-bc25-35924dc6c8d4 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:30.538675 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 864/1733] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:29 2022] POST /identity/v3/users => generated 293 bytes in 779 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:25:30.544086 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef94867-0190-4ab0-bf39-8cbdcd0ce163 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:30.571596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef94867-0190-4ab0-bf39-8cbdcd0ce163 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:30.598625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeba2d8f-cfd8-478d-8570-55e0cae4c27f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:30.599272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeba2d8f-cfd8-478d-8570-55e0cae4c27f admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:30.599545 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeba2d8f-cfd8-478d-8570-55e0cae4c27f admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:30.599915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeba2d8f-cfd8-478d-8570-55e0cae4c27f admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:30.603866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeba2d8f-cfd8-478d-8570-55e0cae4c27f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:30.610857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeba2d8f-cfd8-478d-8570-55e0cae4c27f admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:30.639877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 870/1734] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:30 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:30.644670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d520e4d7-e3b0-4624-bc25-35924dc6c8d4 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:30.665209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d520e4d7-e3b0-4624-bc25-35924dc6c8d4 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:30.696957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1732ac60-b9e2-4d71-87da-7cc573f8f94a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:30.697878 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1732ac60-b9e2-4d71-87da-7cc573f8f94a admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:30.698155 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1732ac60-b9e2-4d71-87da-7cc573f8f94a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:30.698476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1732ac60-b9e2-4d71-87da-7cc573f8f94a admin admin] PATH_INFO: `/v3/projects/7c0530688bca4b90aaf9906b4e61fd0e/users/f6bfd86faf4b4bbcbc95437da09fc324/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:30.742071 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1732ac60-b9e2-4d71-87da-7cc573f8f94a admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c0530688bca4b90aaf9906b4e61fd0e, user_id=f6bfd86faf4b4bbcbc95437da09fc324, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:30.745654 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1732ac60-b9e2-4d71-87da-7cc573f8f94a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:30.764916 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd99df22-bb0f-4454-baaf-346adc957b92 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:30.765394 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd99df22-bb0f-4454-baaf-346adc957b92 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:30.765817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd99df22-bb0f-4454-baaf-346adc957b92 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:30.769440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 871/1735] 10.222.0.47 () {54 vars in 921 bytes} [Sun Aug 21 01:25:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:25:30.850012 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25086bc6-23bd-4a9c-875a-e8c478f852e2 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:30.850296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25086bc6-23bd-4a9c-875a-e8c478f852e2 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:30.850651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25086bc6-23bd-4a9c-875a-e8c478f852e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:30.884301 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 865/1736] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:30 2022] PUT /identity/v3/projects/7c0530688bca4b90aaf9906b4e61fd0e/users/f6bfd86faf4b4bbcbc95437da09fc324/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:30.886231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1732ac60-b9e2-4d71-87da-7cc573f8f94a admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:30.937669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-25086bc6-23bd-4a9c-875a-e8c478f852e2 None None] MFA Rules not processed for user `fabdcdc25d1d4ef6bc66bd6cadecaaf8`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:30.939385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-25086bc6-23bd-4a9c-875a-e8c478f852e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:30.978933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:31.055512 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1732ac60-b9e2-4d71-87da-7cc573f8f94a admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:31.061931 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a38071e-9411-4039-9fa9-643ec6b28538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:31.063133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a38071e-9411-4039-9fa9-643ec6b28538 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:31.064093 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a38071e-9411-4039-9fa9-643ec6b28538 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:31.064624 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a38071e-9411-4039-9fa9-643ec6b28538 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:31.065730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a38071e-9411-4039-9fa9-643ec6b28538 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:31.068021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a38071e-9411-4039-9fa9-643ec6b28538 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:31.103510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 866/1737] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:30 2022] GET /identity/v3/roles => generated 1420 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:31.113141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a38071e-9411-4039-9fa9-643ec6b28538 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:31.141980 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a38071e-9411-4039-9fa9-643ec6b28538 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:31.166849 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243b9bb6-2363-474e-9b02-08123560a27b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:31.168446 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243b9bb6-2363-474e-9b02-08123560a27b admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:31.168728 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243b9bb6-2363-474e-9b02-08123560a27b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:31.169002 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243b9bb6-2363-474e-9b02-08123560a27b admin admin] PATH_INFO: `/v3/projects/7c0530688bca4b90aaf9906b4e61fd0e/users/f6bfd86faf4b4bbcbc95437da09fc324/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:31.180040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243b9bb6-2363-474e-9b02-08123560a27b admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c0530688bca4b90aaf9906b4e61fd0e, user_id=f6bfd86faf4b4bbcbc95437da09fc324, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:31.189758 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243b9bb6-2363-474e-9b02-08123560a27b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:31.219666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 867/1738] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:31 2022] PUT /identity/v3/projects/7c0530688bca4b90aaf9906b4e61fd0e/users/f6bfd86faf4b4bbcbc95437da09fc324/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:31.250707 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1002ce-f7e6-4e99-a6a2-9279860b4ca5 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:31.251051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1002ce-f7e6-4e99-a6a2-9279860b4ca5 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:31.251324 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1002ce-f7e6-4e99-a6a2-9279860b4ca5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:31.281807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 872/1739] 10.222.0.47 () {58 vars in 1018 bytes} [Sun Aug 21 01:25:30 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 428 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:31.343416 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-1a1002ce-f7e6-4e99-a6a2-9279860b4ca5 None None] MFA Rules not processed for user `f6bfd86faf4b4bbcbc95437da09fc324`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:31.345360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-1a1002ce-f7e6-4e99-a6a2-9279860b4ca5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:31.557956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 868/1740] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:31 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:31.572919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25086bc6-23bd-4a9c-875a-e8c478f852e2 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:31.574472 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-25086bc6-23bd-4a9c-875a-e8c478f852e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:31.583899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1002ce-f7e6-4e99-a6a2-9279860b4ca5 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:31.586569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-1a1002ce-f7e6-4e99-a6a2-9279860b4ca5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:31.819607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1002ce-f7e6-4e99-a6a2-9279860b4ca5 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:31.819738 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25086bc6-23bd-4a9c-875a-e8c478f852e2 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:31.832341 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c1d6a5-095f-4dca-9e2d-7b327c89a642 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:31.833329 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c1d6a5-095f-4dca-9e2d-7b327c89a642 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:31.833788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c1d6a5-095f-4dca-9e2d-7b327c89a642 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:31.834783 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c1d6a5-095f-4dca-9e2d-7b327c89a642 admin admin] PATH_INFO: `/v3/users/b1965136d19e4fb4b13c9fd060c77635` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:31.836827 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c1d6a5-095f-4dca-9e2d-7b327c89a642 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b1965136d19e4fb4b13c9fd060c77635)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:31.839035 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c1d6a5-095f-4dca-9e2d-7b327c89a642 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:32.010395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9bf997-452c-4089-85cb-9170268572a6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:32.025032 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9bf997-452c-4089-85cb-9170268572a6 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:32.025032 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9bf997-452c-4089-85cb-9170268572a6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:32.025032 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9bf997-452c-4089-85cb-9170268572a6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:32.041318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9bf997-452c-4089-85cb-9170268572a6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:32.043181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9bf997-452c-4089-85cb-9170268572a6 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:32.063958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 873/1741] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:31 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 490 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:25:32.067716 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9bf997-452c-4089-85cb-9170268572a6 service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:32.073095 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-0a9bf997-452c-4089-85cb-9170268572a6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:32.273701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9bf997-452c-4089-85cb-9170268572a6 service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:32.284256 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b7c0ab-66af-465d-bf5b-ad9dfdb2dfa4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:32.285456 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b7c0ab-66af-465d-bf5b-ad9dfdb2dfa4 service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:32.285746 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b7c0ab-66af-465d-bf5b-ad9dfdb2dfa4 service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:32.286087 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b7c0ab-66af-465d-bf5b-ad9dfdb2dfa4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:32.337752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b7c0ab-66af-465d-bf5b-ad9dfdb2dfa4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:32.339591 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b7c0ab-66af-465d-bf5b-ad9dfdb2dfa4 service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:32.370073 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 874/1742] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:32.462090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-55c1d6a5-095f-4dca-9e2d-7b327c89a642 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b1965136d19e4fb4b13c9fd060c77635'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:32.483298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-55c1d6a5-095f-4dca-9e2d-7b327c89a642 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b1965136d19e4fb4b13c9fd060c77635'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:32.484533 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-55c1d6a5-095f-4dca-9e2d-7b327c89a642 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b1965136d19e4fb4b13c9fd060c77635'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:32.505477 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-55c1d6a5-095f-4dca-9e2d-7b327c89a642 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b1965136d19e4fb4b13c9fd060c77635'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:32.555545 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c426e45a-b45b-437e-8784-0c984ad220fa None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:32.556084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c426e45a-b45b-437e-8784-0c984ad220fa None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:32.556544 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c426e45a-b45b-437e-8784-0c984ad220fa None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:32.557927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 875/1743] 10.222.0.47 () {54 vars in 921 bytes} [Sun Aug 21 01:25:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:25:32.596647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc43aec-479b-46de-9330-8861ccb788f3 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:32.603431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc43aec-479b-46de-9330-8861ccb788f3 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:32.603431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc43aec-479b-46de-9330-8861ccb788f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:32.622055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-cbc43aec-479b-46de-9330-8861ccb788f3 None None] MFA Rules not processed for user `fabdcdc25d1d4ef6bc66bd6cadecaaf8`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:32.623374 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-cbc43aec-479b-46de-9330-8861ccb788f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:32.627488 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-55c1d6a5-095f-4dca-9e2d-7b327c89a642 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b1965136d19e4fb4b13c9fd060c77635'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:32.634438 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 869/1744] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:31 2022] DELETE /identity/v3/users/b1965136d19e4fb4b13c9fd060c77635 => generated 0 bytes in 1051 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:32.637853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 876/1745] 10.222.0.47 () {58 vars in 1018 bytes} [Sun Aug 21 01:25:32 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:25:32.639004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694d3d04-bd7a-4b24-8c67-2b61f362520f None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:32.639396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694d3d04-bd7a-4b24-8c67-2b61f362520f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:32.639784 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694d3d04-bd7a-4b24-8c67-2b61f362520f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:32.663305 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-694d3d04-bd7a-4b24-8c67-2b61f362520f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:32.664989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-694d3d04-bd7a-4b24-8c67-2b61f362520f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:32.757379 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 870/1746] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:32.763486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc43aec-479b-46de-9330-8861ccb788f3 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:32.765092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-cbc43aec-479b-46de-9330-8861ccb788f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:32.839087 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc43aec-479b-46de-9330-8861ccb788f3 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:32.848290 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:32.957818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694d3d04-bd7a-4b24-8c67-2b61f362520f None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:32.960018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb665b2-1d2f-46d9-94d9-0b22240e8cec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:32.960594 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-694d3d04-bd7a-4b24-8c67-2b61f362520f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:32.960707 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb665b2-1d2f-46d9-94d9-0b22240e8cec service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:32.960942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb665b2-1d2f-46d9-94d9-0b22240e8cec service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:32.961211 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb665b2-1d2f-46d9-94d9-0b22240e8cec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:32.983547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb665b2-1d2f-46d9-94d9-0b22240e8cec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:32.985164 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb665b2-1d2f-46d9-94d9-0b22240e8cec service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:32.999214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:33.010564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 877/1747] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:33.138439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694d3d04-bd7a-4b24-8c67-2b61f362520f None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:33.144339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2c1ced-4d35-4d52-bce7-d43ca49507ca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:33.144962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2c1ced-4d35-4d52-bce7-d43ca49507ca service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:33.145240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2c1ced-4d35-4d52-bce7-d43ca49507ca service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:33.145519 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2c1ced-4d35-4d52-bce7-d43ca49507ca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:33.156662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2c1ced-4d35-4d52-bce7-d43ca49507ca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:33.158374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2c1ced-4d35-4d52-bce7-d43ca49507ca service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:33.177784 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 871/1748] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:25:33.734272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb665b2-1d2f-46d9-94d9-0b22240e8cec service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:33.734929 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2c1ced-4d35-4d52-bce7-d43ca49507ca service nova] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:33.735883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-cfb665b2-1d2f-46d9-94d9-0b22240e8cec service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:33.736420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-0a2c1ced-4d35-4d52-bce7-d43ca49507ca service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:33.751083 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb665b2-1d2f-46d9-94d9-0b22240e8cec service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:33.752180 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2c1ced-4d35-4d52-bce7-d43ca49507ca service nova] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:33.756544 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f618711f-85fa-47f3-b2f4-ec5b752b58e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:33.757964 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f618711f-85fa-47f3-b2f4-ec5b752b58e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:33.758339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7d789a-5d6e-40d8-876b-e962d1061f41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:33.758534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f618711f-85fa-47f3-b2f4-ec5b752b58e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:33.761156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f618711f-85fa-47f3-b2f4-ec5b752b58e9 admin admin] PATH_INFO: `/v3/projects/6df4d952c8ac4683a07161bc62dbdba5` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:33.763307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7d789a-5d6e-40d8-876b-e962d1061f41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:33.763506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7d789a-5d6e-40d8-876b-e962d1061f41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:33.763795 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7d789a-5d6e-40d8-876b-e962d1061f41 admin admin] PATH_INFO: `/v3/users/c202efb38590421b970f80f9d693122f` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:33.765103 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f618711f-85fa-47f3-b2f4-ec5b752b58e9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6df4d952c8ac4683a07161bc62dbdba5)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:33.765232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7d789a-5d6e-40d8-876b-e962d1061f41 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c202efb38590421b970f80f9d693122f)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:33.768914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f618711f-85fa-47f3-b2f4-ec5b752b58e9 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:33.783079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7d789a-5d6e-40d8-876b-e962d1061f41 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:34.204240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-af7d789a-5d6e-40d8-876b-e962d1061f41 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c202efb38590421b970f80f9d693122f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.206706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f618711f-85fa-47f3-b2f4-ec5b752b58e9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6df4d952c8ac4683a07161bc62dbdba5', 'request_id': 'req-f618711f-85fa-47f3-b2f4-ec5b752b58e9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.214038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-af7d789a-5d6e-40d8-876b-e962d1061f41 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c202efb38590421b970f80f9d693122f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.214806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-af7d789a-5d6e-40d8-876b-e962d1061f41 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c202efb38590421b970f80f9d693122f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.229778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-af7d789a-5d6e-40d8-876b-e962d1061f41 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c202efb38590421b970f80f9d693122f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.335948 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f618711f-85fa-47f3-b2f4-ec5b752b58e9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6df4d952c8ac4683a07161bc62dbdba5', 'request_id': 'req-f618711f-85fa-47f3-b2f4-ec5b752b58e9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.342363 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f618711f-85fa-47f3-b2f4-ec5b752b58e9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6df4d952c8ac4683a07161bc62dbdba5', 'request_id': 'req-f618711f-85fa-47f3-b2f4-ec5b752b58e9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.360692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f618711f-85fa-47f3-b2f4-ec5b752b58e9 admin admin] The token cache is being invalidate because project 6df4d952c8ac4683a07161bc62dbdba5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:34.361038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f618711f-85fa-47f3-b2f4-ec5b752b58e9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.361788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f618711f-85fa-47f3-b2f4-ec5b752b58e9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.363425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 878/1749] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:33 2022] DELETE /identity/v3/projects/6df4d952c8ac4683a07161bc62dbdba5 => generated 0 bytes in 630 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:34.365450 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f618711f-85fa-47f3-b2f4-ec5b752b58e9 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:34.367544 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-f618711f-85fa-47f3-b2f4-ec5b752b58e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:34.403028 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:34.433994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f618711f-85fa-47f3-b2f4-ec5b752b58e9 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:34.438831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e227ab50-021e-4e47-b357-1bf457bad019 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:34.439452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e227ab50-021e-4e47-b357-1bf457bad019 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:34.439735 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e227ab50-021e-4e47-b357-1bf457bad019 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:34.440003 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e227ab50-021e-4e47-b357-1bf457bad019 admin admin] PATH_INFO: `/v3/users/2787f295dc064a9bb96b70bd651df3d8` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:34.441150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e227ab50-021e-4e47-b357-1bf457bad019 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2787f295dc064a9bb96b70bd651df3d8)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:34.443117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e227ab50-021e-4e47-b357-1bf457bad019 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:34.487906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-af7d789a-5d6e-40d8-876b-e962d1061f41 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c202efb38590421b970f80f9d693122f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.496507 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 872/1750] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:33 2022] DELETE /identity/v3/users/c202efb38590421b970f80f9d693122f => generated 0 bytes in 761 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:34.496507 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7d789a-5d6e-40d8-876b-e962d1061f41 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:34.497150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-af7d789a-5d6e-40d8-876b-e962d1061f41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:34.566806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7d789a-5d6e-40d8-876b-e962d1061f41 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:34.571608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a412d9d-940c-44ca-ba4e-9530d475089a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:34.572302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a412d9d-940c-44ca-ba4e-9530d475089a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:34.572583 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a412d9d-940c-44ca-ba4e-9530d475089a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:34.572856 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a412d9d-940c-44ca-ba4e-9530d475089a admin admin] PATH_INFO: `/v3/users/26620316dced41389244d395eea2b037` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:34.574125 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a412d9d-940c-44ca-ba4e-9530d475089a admin admin] RBAC: Authorizing `identity:delete_user(user_id=26620316dced41389244d395eea2b037)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:34.576186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a412d9d-940c-44ca-ba4e-9530d475089a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:34.815799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e227ab50-021e-4e47-b357-1bf457bad019 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2787f295dc064a9bb96b70bd651df3d8'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.823268 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e227ab50-021e-4e47-b357-1bf457bad019 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2787f295dc064a9bb96b70bd651df3d8'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.836478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e227ab50-021e-4e47-b357-1bf457bad019 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2787f295dc064a9bb96b70bd651df3d8'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.916002 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1a412d9d-940c-44ca-ba4e-9530d475089a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26620316dced41389244d395eea2b037'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.923427 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1a412d9d-940c-44ca-ba4e-9530d475089a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26620316dced41389244d395eea2b037'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.924502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1a412d9d-940c-44ca-ba4e-9530d475089a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26620316dced41389244d395eea2b037'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.930367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e227ab50-021e-4e47-b357-1bf457bad019 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2787f295dc064a9bb96b70bd651df3d8'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.931045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e227ab50-021e-4e47-b357-1bf457bad019 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2787f295dc064a9bb96b70bd651df3d8'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.938591 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 879/1751] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:34 2022] DELETE /identity/v3/users/2787f295dc064a9bb96b70bd651df3d8 => generated 0 bytes in 575 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:34.944347 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa9fb70-104a-4e58-b173-7ae73d0bfa39 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:34.945144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa9fb70-104a-4e58-b173-7ae73d0bfa39 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:34.945865 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa9fb70-104a-4e58-b173-7ae73d0bfa39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:34.947027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1a412d9d-940c-44ca-ba4e-9530d475089a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26620316dced41389244d395eea2b037'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:34.965956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-0fa9fb70-104a-4e58-b173-7ae73d0bfa39 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:34.970216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-0fa9fb70-104a-4e58-b173-7ae73d0bfa39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:35.025245 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1a412d9d-940c-44ca-ba4e-9530d475089a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26620316dced41389244d395eea2b037'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:35.037361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 873/1752] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:34 2022] DELETE /identity/v3/users/26620316dced41389244d395eea2b037 => generated 0 bytes in 538 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:35.037361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cd219b-dab2-45e1-ad2a-386cfaf89f4e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:35.037361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cd219b-dab2-45e1-ad2a-386cfaf89f4e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:35.037361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cd219b-dab2-45e1-ad2a-386cfaf89f4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:35.047544 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 880/1753] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:35.063900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa9fb70-104a-4e58-b173-7ae73d0bfa39 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:35.071484 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-0fa9fb70-104a-4e58-b173-7ae73d0bfa39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:35.081402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-97cd219b-dab2-45e1-ad2a-386cfaf89f4e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:35.082850 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-97cd219b-dab2-45e1-ad2a-386cfaf89f4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:35.164029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:35.165722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa9fb70-104a-4e58-b173-7ae73d0bfa39 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:35.168539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 874/1754] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:35.182228 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cd219b-dab2-45e1-ad2a-386cfaf89f4e None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:35.271564 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cd219b-dab2-45e1-ad2a-386cfaf89f4e None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:35.277122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818f6202-c7a4-48e8-837b-60dce0c5cf68 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:35.277826 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818f6202-c7a4-48e8-837b-60dce0c5cf68 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:35.278109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818f6202-c7a4-48e8-837b-60dce0c5cf68 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:35.278428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818f6202-c7a4-48e8-837b-60dce0c5cf68 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:35.294881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818f6202-c7a4-48e8-837b-60dce0c5cf68 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:35.296449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6a3a2c-fb59-49fc-95f5-557d4a465499 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:35.296844 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818f6202-c7a4-48e8-837b-60dce0c5cf68 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:35.297568 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6a3a2c-fb59-49fc-95f5-557d4a465499 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:35.298078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6a3a2c-fb59-49fc-95f5-557d4a465499 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:35.298625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6a3a2c-fb59-49fc-95f5-557d4a465499 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:35.320851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6a3a2c-fb59-49fc-95f5-557d4a465499 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:35.322792 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6a3a2c-fb59-49fc-95f5-557d4a465499 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:35.327764 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 875/1755] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:35.344193 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 881/1756] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 281 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:36.571550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818f6202-c7a4-48e8-837b-60dce0c5cf68 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:36.571550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-818f6202-c7a4-48e8-837b-60dce0c5cf68 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:36.591189 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818f6202-c7a4-48e8-837b-60dce0c5cf68 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:36.599259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:36.600726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:36.601199 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:36.601549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc admin admin] PATH_INFO: `/v3/projects/e46a7357852c41d9b794764bff4c2b41` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:36.602890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc admin admin] RBAC: Authorizing `identity:delete_project(project_id=e46a7357852c41d9b794764bff4c2b41)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:36.619163 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:36.689074 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6a3a2c-fb59-49fc-95f5-557d4a465499 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:36.689074 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9c6a3a2c-fb59-49fc-95f5-557d4a465499 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:36.704710 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6a3a2c-fb59-49fc-95f5-557d4a465499 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:36.710914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73da4a6-6e97-4ef1-bf9f-6692e7573e67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:36.711576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73da4a6-6e97-4ef1-bf9f-6692e7573e67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:36.711853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73da4a6-6e97-4ef1-bf9f-6692e7573e67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:36.712153 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73da4a6-6e97-4ef1-bf9f-6692e7573e67 admin admin] PATH_INFO: `/v3/projects/b009d4c2e5f64d8b8b6b5c702067f8bf` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:36.713372 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73da4a6-6e97-4ef1-bf9f-6692e7573e67 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b009d4c2e5f64d8b8b6b5c702067f8bf)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:36.715333 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73da4a6-6e97-4ef1-bf9f-6692e7573e67 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:36.848141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e46a7357852c41d9b794764bff4c2b41', 'request_id': 'req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:36.857951 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e46a7357852c41d9b794764bff4c2b41', 'request_id': 'req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:36.867273 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e46a7357852c41d9b794764bff4c2b41', 'request_id': 'req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:36.905621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d73da4a6-6e97-4ef1-bf9f-6692e7573e67 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b009d4c2e5f64d8b8b6b5c702067f8bf', 'request_id': 'req-d73da4a6-6e97-4ef1-bf9f-6692e7573e67'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:36.971998 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc admin admin] The token cache is being invalidate because project e46a7357852c41d9b794764bff4c2b41 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:36.982941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:36.982941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:36.982941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 876/1757] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:36 2022] DELETE /identity/v3/projects/e46a7357852c41d9b794764bff4c2b41 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:36.982941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:36.982941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:37.057549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dccf0a-2b6c-4c60-b2ee-ab82171109cc admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:37.124540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:37.125208 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:37.125490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:37.125763 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8 admin admin] PATH_INFO: `/v3/projects/9700cc0520bb44f59dcede462a56bcbc` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:37.127185 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9700cc0520bb44f59dcede462a56bcbc)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:37.129198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:37.258855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d73da4a6-6e97-4ef1-bf9f-6692e7573e67 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b009d4c2e5f64d8b8b6b5c702067f8bf', 'request_id': 'req-d73da4a6-6e97-4ef1-bf9f-6692e7573e67'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:37.272064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d73da4a6-6e97-4ef1-bf9f-6692e7573e67 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b009d4c2e5f64d8b8b6b5c702067f8bf', 'request_id': 'req-d73da4a6-6e97-4ef1-bf9f-6692e7573e67'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:37.294156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d73da4a6-6e97-4ef1-bf9f-6692e7573e67 admin admin] The token cache is being invalidate because project b009d4c2e5f64d8b8b6b5c702067f8bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:37.294798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d73da4a6-6e97-4ef1-bf9f-6692e7573e67 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:37.295845 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d73da4a6-6e97-4ef1-bf9f-6692e7573e67 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:37.299276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 882/1758] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:36 2022] DELETE /identity/v3/projects/b009d4c2e5f64d8b8b6b5c702067f8bf => generated 0 bytes in 622 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:25:37.301741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a34142-c6af-4171-8629-e7461dd68471 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:37.302802 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a34142-c6af-4171-8629-e7461dd68471 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:37.303416 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a34142-c6af-4171-8629-e7461dd68471 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:37.344523 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9700cc0520bb44f59dcede462a56bcbc', 'request_id': 'req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:37.345321 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-76a34142-c6af-4171-8629-e7461dd68471 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:37.347857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-76a34142-c6af-4171-8629-e7461dd68471 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:37.360615 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9700cc0520bb44f59dcede462a56bcbc', 'request_id': 'req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:37.375762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9700cc0520bb44f59dcede462a56bcbc', 'request_id': 'req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:37.455186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8 admin admin] The token cache is being invalidate because project 9700cc0520bb44f59dcede462a56bcbc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:37.455485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:37.459679 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:37.461625 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 877/1759] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:36 2022] DELETE /identity/v3/projects/9700cc0520bb44f59dcede462a56bcbc => generated 0 bytes in 482 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:37.462594 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:37.464158 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:37.484066 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 883/1760] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:37.484594 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a34142-c6af-4171-8629-e7461dd68471 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:37.506533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-76a34142-c6af-4171-8629-e7461dd68471 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:37.525887 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a34142-c6af-4171-8629-e7461dd68471 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:37.531599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a76c15-7132-493c-b86c-6493f9df4eec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:37.532533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a76c15-7132-493c-b86c-6493f9df4eec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:37.532924 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a76c15-7132-493c-b86c-6493f9df4eec admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:37.533186 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a76c15-7132-493c-b86c-6493f9df4eec admin admin] PATH_INFO: `/v3/users/3852478ed0ce4b49b1f7677a962b9453` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:37.534614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a76c15-7132-493c-b86c-6493f9df4eec admin admin] RBAC: Authorizing `identity:delete_user(user_id=3852478ed0ce4b49b1f7677a962b9453)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:37.537664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a76c15-7132-493c-b86c-6493f9df4eec admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:37.545841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65bacb6d-8e6a-4aa0-9e36-62d7c8dfd0d8 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:37.553278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a59018a-2af0-421c-bd16-be3e994ef5e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:37.554241 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a59018a-2af0-421c-bd16-be3e994ef5e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:37.554705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a59018a-2af0-421c-bd16-be3e994ef5e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:37.555145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a59018a-2af0-421c-bd16-be3e994ef5e7 admin admin] PATH_INFO: `/v3/users/f6bfd86faf4b4bbcbc95437da09fc324` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:37.557265 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a59018a-2af0-421c-bd16-be3e994ef5e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f6bfd86faf4b4bbcbc95437da09fc324)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:37.559962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a59018a-2af0-421c-bd16-be3e994ef5e7 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:38.059203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e3a76c15-7132-493c-b86c-6493f9df4eec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3852478ed0ce4b49b1f7677a962b9453'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:38.085265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e3a76c15-7132-493c-b86c-6493f9df4eec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3852478ed0ce4b49b1f7677a962b9453'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:38.095262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0a59018a-2af0-421c-bd16-be3e994ef5e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f6bfd86faf4b4bbcbc95437da09fc324'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:38.103161 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0a59018a-2af0-421c-bd16-be3e994ef5e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f6bfd86faf4b4bbcbc95437da09fc324'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:38.103983 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0a59018a-2af0-421c-bd16-be3e994ef5e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f6bfd86faf4b4bbcbc95437da09fc324'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:38.109209 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e3a76c15-7132-493c-b86c-6493f9df4eec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3852478ed0ce4b49b1f7677a962b9453'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:38.120638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0a59018a-2af0-421c-bd16-be3e994ef5e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f6bfd86faf4b4bbcbc95437da09fc324'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:38.253676 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e3a76c15-7132-493c-b86c-6493f9df4eec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3852478ed0ce4b49b1f7677a962b9453'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:38.254622 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e3a76c15-7132-493c-b86c-6493f9df4eec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3852478ed0ce4b49b1f7677a962b9453'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:38.266685 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 884/1761] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:37 2022] DELETE /identity/v3/users/3852478ed0ce4b49b1f7677a962b9453 => generated 0 bytes in 782 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:38.275413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da2e08e-ff8a-4db0-a9fe-401420084419 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:38.275413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da2e08e-ff8a-4db0-a9fe-401420084419 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:38.275413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da2e08e-ff8a-4db0-a9fe-401420084419 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:38.337332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-2da2e08e-ff8a-4db0-a9fe-401420084419 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:38.340989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-2da2e08e-ff8a-4db0-a9fe-401420084419 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:38.482387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 885/1762] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:38.483745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da2e08e-ff8a-4db0-a9fe-401420084419 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:38.499334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da2e08e-ff8a-4db0-a9fe-401420084419 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:38.507665 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f95fd1-3c95-449b-852f-a52d2419b297 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:38.508335 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f95fd1-3c95-449b-852f-a52d2419b297 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:38.508766 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f95fd1-3c95-449b-852f-a52d2419b297 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:38.511622 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f95fd1-3c95-449b-852f-a52d2419b297 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:38.512527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f95fd1-3c95-449b-852f-a52d2419b297 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:25:38.513057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f95fd1-3c95-449b-852f-a52d2419b297 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:38.515375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f95fd1-3c95-449b-852f-a52d2419b297 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:38.532535 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 886/1763] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:25:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:25:38.535977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f95fd1-3c95-449b-852f-a52d2419b297 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:38.549593 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f95fd1-3c95-449b-852f-a52d2419b297 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:38.554545 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0a59018a-2af0-421c-bd16-be3e994ef5e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f6bfd86faf4b4bbcbc95437da09fc324'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:38.561431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696c4f4a-07d4-427b-80b9-33f03cd18a41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:38.561431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696c4f4a-07d4-427b-80b9-33f03cd18a41 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:38.561431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696c4f4a-07d4-427b-80b9-33f03cd18a41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:38.561431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696c4f4a-07d4-427b-80b9-33f03cd18a41 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:38.561841 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696c4f4a-07d4-427b-80b9-33f03cd18a41 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:38.561841 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696c4f4a-07d4-427b-80b9-33f03cd18a41 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:38.564444 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 878/1764] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:37 2022] DELETE /identity/v3/users/f6bfd86faf4b4bbcbc95437da09fc324 => generated 0 bytes in 1102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:38.830548 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 887/1765] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:38 2022] POST /identity/v3/projects => generated 377 bytes in 295 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:25:38.837487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a59018a-2af0-421c-bd16-be3e994ef5e7 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:38.840672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-0a59018a-2af0-421c-bd16-be3e994ef5e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:38.991583 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a59018a-2af0-421c-bd16-be3e994ef5e7 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:39.100842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1613439a-28ec-43c1-b723-99f48663a48c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:39.105390 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1613439a-28ec-43c1-b723-99f48663a48c admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:39.105835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1613439a-28ec-43c1-b723-99f48663a48c admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:39.106100 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1613439a-28ec-43c1-b723-99f48663a48c admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:39.106915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1613439a-28ec-43c1-b723-99f48663a48c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:39.109236 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1613439a-28ec-43c1-b723-99f48663a48c admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:39.242022 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 879/1766] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:38 2022] POST /identity/v3/users => generated 289 bytes in 406 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:39.247327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696c4f4a-07d4-427b-80b9-33f03cd18a41 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:39.268847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696c4f4a-07d4-427b-80b9-33f03cd18a41 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:39.296732 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c624ae6b-5feb-40b7-8a20-d5efa4cd53f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:39.300221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c624ae6b-5feb-40b7-8a20-d5efa4cd53f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:39.300615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c624ae6b-5feb-40b7-8a20-d5efa4cd53f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:39.301043 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c624ae6b-5feb-40b7-8a20-d5efa4cd53f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:39.302124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c624ae6b-5feb-40b7-8a20-d5efa4cd53f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:39.304272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c624ae6b-5feb-40b7-8a20-d5efa4cd53f4 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:39.327663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 888/1767] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:39 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:39.332573 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1613439a-28ec-43c1-b723-99f48663a48c admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:39.369443 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1613439a-28ec-43c1-b723-99f48663a48c admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:39.375850 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09dbbc5c-f3f3-4a1a-bbff-f13aa4add459 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:39.376767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dbbc5c-f3f3-4a1a-bbff-f13aa4add459 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:39.376953 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dbbc5c-f3f3-4a1a-bbff-f13aa4add459 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:39.377135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dbbc5c-f3f3-4a1a-bbff-f13aa4add459 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:39.378255 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09dbbc5c-f3f3-4a1a-bbff-f13aa4add459 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:39.380191 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09dbbc5c-f3f3-4a1a-bbff-f13aa4add459 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:39.393336 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 880/1768] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:39 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:39.405156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c624ae6b-5feb-40b7-8a20-d5efa4cd53f4 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:39.417947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c624ae6b-5feb-40b7-8a20-d5efa4cd53f4 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:39.424195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e2dbdf-c2cd-4e0b-ba12-b5b05e668bea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:39.424967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e2dbdf-c2cd-4e0b-ba12-b5b05e668bea admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:39.425635 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e2dbdf-c2cd-4e0b-ba12-b5b05e668bea admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:39.426299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e2dbdf-c2cd-4e0b-ba12-b5b05e668bea admin admin] PATH_INFO: `/v3/projects/063b0c2f3f7942cf95d5cd5ee6c509ef/users/ef1b0ba55baa4805b8b21fcd86e801e1/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:39.436453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09dbbc5c-f3f3-4a1a-bbff-f13aa4add459 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:39.441332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-09dbbc5c-f3f3-4a1a-bbff-f13aa4add459 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:39.448718 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e2dbdf-c2cd-4e0b-ba12-b5b05e668bea admin admin] RBAC: Authorizing `identity:create_grant(project_id=063b0c2f3f7942cf95d5cd5ee6c509ef, user_id=ef1b0ba55baa4805b8b21fcd86e801e1, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:39.452807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e2dbdf-c2cd-4e0b-ba12-b5b05e668bea admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:39.467111 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:39.470209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09dbbc5c-f3f3-4a1a-bbff-f13aa4add459 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:39.477586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea736c42-99cd-4f88-b73e-cf6e65c237d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:39.478252 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea736c42-99cd-4f88-b73e-cf6e65c237d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:39.478532 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea736c42-99cd-4f88-b73e-cf6e65c237d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:39.479061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea736c42-99cd-4f88-b73e-cf6e65c237d1 admin admin] PATH_INFO: `/v3/projects/7c0530688bca4b90aaf9906b4e61fd0e` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:39.480654 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea736c42-99cd-4f88-b73e-cf6e65c237d1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7c0530688bca4b90aaf9906b4e61fd0e)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:39.483713 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea736c42-99cd-4f88-b73e-cf6e65c237d1 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:39.530946 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 889/1769] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:39 2022] PUT /identity/v3/projects/063b0c2f3f7942cf95d5cd5ee6c509ef/users/ef1b0ba55baa4805b8b21fcd86e801e1/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:39.542448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e2dbdf-c2cd-4e0b-ba12-b5b05e668bea admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:39.547201 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-20e2dbdf-c2cd-4e0b-ba12-b5b05e668bea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:39.658938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:39.665672 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e2dbdf-c2cd-4e0b-ba12-b5b05e668bea admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:39.688395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dd511a-175a-4fae-a304-10143472cc6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:39.690893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dd511a-175a-4fae-a304-10143472cc6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:39.691295 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dd511a-175a-4fae-a304-10143472cc6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:39.691967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dd511a-175a-4fae-a304-10143472cc6b admin admin] PATH_INFO: `/v3/projects/e3d46a17d2444744a6200cbca7471fc3` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:39.695561 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28dd511a-175a-4fae-a304-10143472cc6b admin admin] RBAC: Authorizing `identity:delete_project(project_id=e3d46a17d2444744a6200cbca7471fc3)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:39.704717 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28dd511a-175a-4fae-a304-10143472cc6b admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:39.705450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ea736c42-99cd-4f88-b73e-cf6e65c237d1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7c0530688bca4b90aaf9906b4e61fd0e', 'request_id': 'req-ea736c42-99cd-4f88-b73e-cf6e65c237d1'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:39.710976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ea736c42-99cd-4f88-b73e-cf6e65c237d1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7c0530688bca4b90aaf9906b4e61fd0e', 'request_id': 'req-ea736c42-99cd-4f88-b73e-cf6e65c237d1'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:39.738079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ea736c42-99cd-4f88-b73e-cf6e65c237d1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7c0530688bca4b90aaf9906b4e61fd0e', 'request_id': 'req-ea736c42-99cd-4f88-b73e-cf6e65c237d1'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:40.362504 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-28dd511a-175a-4fae-a304-10143472cc6b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e3d46a17d2444744a6200cbca7471fc3', 'request_id': 'req-28dd511a-175a-4fae-a304-10143472cc6b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:40.383297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ea736c42-99cd-4f88-b73e-cf6e65c237d1 admin admin] The token cache is being invalidate because project 7c0530688bca4b90aaf9906b4e61fd0e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:40.386566 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ea736c42-99cd-4f88-b73e-cf6e65c237d1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:40.386566 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ea736c42-99cd-4f88-b73e-cf6e65c237d1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:40.387495 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 881/1770] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:39 2022] DELETE /identity/v3/projects/7c0530688bca4b90aaf9906b4e61fd0e => generated 0 bytes in 952 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:25:40.389156 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfae2ae-89b3-40eb-b19d-bcc818d392bf None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:40.389439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfae2ae-89b3-40eb-b19d-bcc818d392bf None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:40.389711 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfae2ae-89b3-40eb-b19d-bcc818d392bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:40.424944 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-5dfae2ae-89b3-40eb-b19d-bcc818d392bf None None] MFA Rules not processed for user `ef1b0ba55baa4805b8b21fcd86e801e1`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:40.427531 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5dfae2ae-89b3-40eb-b19d-bcc818d392bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:40.526854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 882/1771] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:40 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:40.530207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d0d774-0974-4c0c-b5fa-c7a0215334a4 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:40.530207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d0d774-0974-4c0c-b5fa-c7a0215334a4 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:40.530207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d0d774-0974-4c0c-b5fa-c7a0215334a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:40.573039 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-28dd511a-175a-4fae-a304-10143472cc6b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e3d46a17d2444744a6200cbca7471fc3', 'request_id': 'req-28dd511a-175a-4fae-a304-10143472cc6b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:40.577510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-a5d0d774-0974-4c0c-b5fa-c7a0215334a4 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:40.579646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a5d0d774-0974-4c0c-b5fa-c7a0215334a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:40.582109 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-28dd511a-175a-4fae-a304-10143472cc6b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e3d46a17d2444744a6200cbca7471fc3', 'request_id': 'req-28dd511a-175a-4fae-a304-10143472cc6b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:40.607070 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-28dd511a-175a-4fae-a304-10143472cc6b admin admin] The token cache is being invalidate because project e3d46a17d2444744a6200cbca7471fc3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:40.607452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-28dd511a-175a-4fae-a304-10143472cc6b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:40.609177 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-28dd511a-175a-4fae-a304-10143472cc6b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:40.613486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 890/1772] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:39 2022] DELETE /identity/v3/projects/e3d46a17d2444744a6200cbca7471fc3 => generated 0 bytes in 1082 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:40.614520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dd511a-175a-4fae-a304-10143472cc6b admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:40.621235 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-28dd511a-175a-4fae-a304-10143472cc6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:40.705811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 883/1773] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:40.707684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a97fd40-76f5-455f-8399-d60787376d7e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:40.707989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a97fd40-76f5-455f-8399-d60787376d7e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:40.709185 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a97fd40-76f5-455f-8399-d60787376d7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:40.729212 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dd511a-175a-4fae-a304-10143472cc6b admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:40.737614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630bbcde-6d93-4603-987c-162e95dc2af8 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6365b2328f34ab9b1b73ececbf5cc30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:40.737614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630bbcde-6d93-4603-987c-162e95dc2af8 service glance] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:40.737614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630bbcde-6d93-4603-987c-162e95dc2af8 service glance] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:40.737614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630bbcde-6d93-4603-987c-162e95dc2af8 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:40.738170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-7a97fd40-76f5-455f-8399-d60787376d7e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:40.738269 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-630bbcde-6d93-4603-987c-162e95dc2af8 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:40.738367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7a97fd40-76f5-455f-8399-d60787376d7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:40.745785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 884/1774] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:40.747079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6413a03-2f9e-4a95-9289-7b824db434d7 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:40.747395 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6413a03-2f9e-4a95-9289-7b824db434d7 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:40.747576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630bbcde-6d93-4603-987c-162e95dc2af8 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:40.747669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6413a03-2f9e-4a95-9289-7b824db434d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:40.749052 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630bbcde-6d93-4603-987c-162e95dc2af8 service glance] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:40.759568 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 891/1775] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:40 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:40.760378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5b28cd-899f-4643-ad37-a4a7450764af None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:40.760663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5b28cd-899f-4643-ad37-a4a7450764af None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:40.760968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5b28cd-899f-4643-ad37-a4a7450764af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:40.782292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f6413a03-2f9e-4a95-9289-7b824db434d7 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:40.783665 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f6413a03-2f9e-4a95-9289-7b824db434d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:40.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-2d5b28cd-899f-4643-ad37-a4a7450764af None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:40.785583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-2d5b28cd-899f-4643-ad37-a4a7450764af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:40.797290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 892/1776] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:40.797511 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 885/1777] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:40.798410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5b28cd-899f-4643-ad37-a4a7450764af None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:40.798652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6413a03-2f9e-4a95-9289-7b824db434d7 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:40.809012 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5b28cd-899f-4643-ad37-a4a7450764af None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:40.810969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6413a03-2f9e-4a95-9289-7b824db434d7 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:40.814578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a219ffa-c7f0-47e0-8774-a5c5ad91bfc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:40.822244 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a219ffa-c7f0-47e0-8774-a5c5ad91bfc2 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:40.822244 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a219ffa-c7f0-47e0-8774-a5c5ad91bfc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:40.822244 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a219ffa-c7f0-47e0-8774-a5c5ad91bfc2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:40.822244 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a219ffa-c7f0-47e0-8774-a5c5ad91bfc2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:25:40.822588 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2fb529-4573-447b-ad74-78fa2eabb103 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:40.822687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a219ffa-c7f0-47e0-8774-a5c5ad91bfc2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:40.822765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2fb529-4573-447b-ad74-78fa2eabb103 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:40.822765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2fb529-4573-447b-ad74-78fa2eabb103 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:40.822765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2fb529-4573-447b-ad74-78fa2eabb103 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:40.822961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a219ffa-c7f0-47e0-8774-a5c5ad91bfc2 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:40.823038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2fb529-4573-447b-ad74-78fa2eabb103 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:25:40.823038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2fb529-4573-447b-ad74-78fa2eabb103 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:40.823038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2fb529-4573-447b-ad74-78fa2eabb103 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:40.834348 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 886/1778] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:25:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:25:40.834897 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 893/1779] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:25:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:25:40.838430 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2fb529-4573-447b-ad74-78fa2eabb103 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:40.838581 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a219ffa-c7f0-47e0-8774-a5c5ad91bfc2 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:40.851044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a219ffa-c7f0-47e0-8774-a5c5ad91bfc2 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:40.851149 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2fb529-4573-447b-ad74-78fa2eabb103 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:40.854933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f40e4b2-2697-4176-a357-ceccc0528f90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:40.855728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f40e4b2-2697-4176-a357-ceccc0528f90 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:40.855999 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f40e4b2-2697-4176-a357-ceccc0528f90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:40.856311 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f40e4b2-2697-4176-a357-ceccc0528f90 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:40.856936 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f40e4b2-2697-4176-a357-ceccc0528f90 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:40.859327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f40e4b2-2697-4176-a357-ceccc0528f90 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:40.859457 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c647be75-9c46-463f-b7b1-9c2bbe6afc72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:40.860068 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c647be75-9c46-463f-b7b1-9c2bbe6afc72 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:40.860380 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c647be75-9c46-463f-b7b1-9c2bbe6afc72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:40.860650 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c647be75-9c46-463f-b7b1-9c2bbe6afc72 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:40.861266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c647be75-9c46-463f-b7b1-9c2bbe6afc72 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:40.863308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c647be75-9c46-463f-b7b1-9c2bbe6afc72 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:40.981224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 894/1780] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:40 2022] POST /identity/v3/projects => generated 385 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:25:40.993325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 887/1781] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:40 2022] POST /identity/v3/projects => generated 381 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:25:41.002462 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c647be75-9c46-463f-b7b1-9c2bbe6afc72 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:41.006109 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f40e4b2-2697-4176-a357-ceccc0528f90 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:41.148669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c647be75-9c46-463f-b7b1-9c2bbe6afc72 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:41.161181 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890b1fb0-0811-4a9e-b890-3df5b2b7b47d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:41.162689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890b1fb0-0811-4a9e-b890-3df5b2b7b47d admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:41.162921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890b1fb0-0811-4a9e-b890-3df5b2b7b47d admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:41.165177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890b1fb0-0811-4a9e-b890-3df5b2b7b47d admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:41.165177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890b1fb0-0811-4a9e-b890-3df5b2b7b47d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:41.166991 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890b1fb0-0811-4a9e-b890-3df5b2b7b47d admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:41.194982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f40e4b2-2697-4176-a357-ceccc0528f90 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:41.207899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1deb726d-f246-43dd-8e76-b82619865409 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:41.209325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1deb726d-f246-43dd-8e76-b82619865409 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:41.209760 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1deb726d-f246-43dd-8e76-b82619865409 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:41.210183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1deb726d-f246-43dd-8e76-b82619865409 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:41.210988 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1deb726d-f246-43dd-8e76-b82619865409 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:41.213181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1deb726d-f246-43dd-8e76-b82619865409 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:41.358006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 888/1782] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:41 2022] POST /identity/v3/users => generated 291 bytes in 354 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:41.358176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 895/1783] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:41 2022] POST /identity/v3/users => generated 293 bytes in 353 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:41.359101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890b1fb0-0811-4a9e-b890-3df5b2b7b47d admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:41.360836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1deb726d-f246-43dd-8e76-b82619865409 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:41.369312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890b1fb0-0811-4a9e-b890-3df5b2b7b47d admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:41.370837 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1deb726d-f246-43dd-8e76-b82619865409 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:41.374998 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5899e4f5-cc22-4b59-8d70-b0a6e0b98e91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:41.375711 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5899e4f5-cc22-4b59-8d70-b0a6e0b98e91 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:41.376095 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5899e4f5-cc22-4b59-8d70-b0a6e0b98e91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:41.376486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5899e4f5-cc22-4b59-8d70-b0a6e0b98e91 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:41.376591 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf54e6f-7c9b-4b23-b871-baf5101d8f5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:41.377207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf54e6f-7c9b-4b23-b871-baf5101d8f5d admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:41.377396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5899e4f5-cc22-4b59-8d70-b0a6e0b98e91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:41.377503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf54e6f-7c9b-4b23-b871-baf5101d8f5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:41.377760 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf54e6f-7c9b-4b23-b871-baf5101d8f5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:41.378401 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf54e6f-7c9b-4b23-b871-baf5101d8f5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:41.379368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5899e4f5-cc22-4b59-8d70-b0a6e0b98e91 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:41.380479 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf54e6f-7c9b-4b23-b871-baf5101d8f5d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:41.394442 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 889/1784] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:41 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:41.396570 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5899e4f5-cc22-4b59-8d70-b0a6e0b98e91 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:41.404685 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 896/1785] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:41 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:41.405824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf54e6f-7c9b-4b23-b871-baf5101d8f5d admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:41.409980 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5899e4f5-cc22-4b59-8d70-b0a6e0b98e91 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:41.418559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26a502c-83d1-43db-aba3-9ece2c9d6e1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:41.418559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26a502c-83d1-43db-aba3-9ece2c9d6e1a admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:41.418559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26a502c-83d1-43db-aba3-9ece2c9d6e1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:41.418559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26a502c-83d1-43db-aba3-9ece2c9d6e1a admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:41.419327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26a502c-83d1-43db-aba3-9ece2c9d6e1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:41.419850 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf54e6f-7c9b-4b23-b871-baf5101d8f5d admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:41.420967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26a502c-83d1-43db-aba3-9ece2c9d6e1a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:41.425582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2422a9-33a1-4c6a-954c-aa7b69e76494 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:41.426181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2422a9-33a1-4c6a-954c-aa7b69e76494 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:41.426461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2422a9-33a1-4c6a-954c-aa7b69e76494 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:41.426751 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2422a9-33a1-4c6a-954c-aa7b69e76494 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:41.427394 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2422a9-33a1-4c6a-954c-aa7b69e76494 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:41.429134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2422a9-33a1-4c6a-954c-aa7b69e76494 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:41.432169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 890/1786] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:41 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:25:41.436349 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26a502c-83d1-43db-aba3-9ece2c9d6e1a admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:41.444136 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 897/1787] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:41 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:41.445464 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2422a9-33a1-4c6a-954c-aa7b69e76494 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:41.446887 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26a502c-83d1-43db-aba3-9ece2c9d6e1a admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:41.451491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3346b173-3ae1-4d44-a05c-04e67095a7de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:41.452172 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3346b173-3ae1-4d44-a05c-04e67095a7de admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:41.452451 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3346b173-3ae1-4d44-a05c-04e67095a7de admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:41.452782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3346b173-3ae1-4d44-a05c-04e67095a7de admin admin] PATH_INFO: `/v3/projects/a10ac2feed034f8c84411379d048d9b9/users/691126a73424498ea86d1dcdd8ffe371/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:41.455307 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2422a9-33a1-4c6a-954c-aa7b69e76494 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:41.460882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46892e7-5f2b-4123-ac27-a0bfc0fd6d6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:41.461533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46892e7-5f2b-4123-ac27-a0bfc0fd6d6d admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:41.470663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46892e7-5f2b-4123-ac27-a0bfc0fd6d6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:41.470663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46892e7-5f2b-4123-ac27-a0bfc0fd6d6d admin admin] PATH_INFO: `/v3/projects/e7cc49c6d4ea4cae924c6af52494e6f0/users/a5a5f29fc50e4fdfb8a9fa6fbac48099/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:41.470887 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3346b173-3ae1-4d44-a05c-04e67095a7de admin admin] RBAC: Authorizing `identity:create_grant(project_id=a10ac2feed034f8c84411379d048d9b9, user_id=691126a73424498ea86d1dcdd8ffe371, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:41.472618 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3346b173-3ae1-4d44-a05c-04e67095a7de admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:41.476486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46892e7-5f2b-4123-ac27-a0bfc0fd6d6d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e7cc49c6d4ea4cae924c6af52494e6f0, user_id=a5a5f29fc50e4fdfb8a9fa6fbac48099, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:41.479260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46892e7-5f2b-4123-ac27-a0bfc0fd6d6d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:41.588033 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 891/1788] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:41 2022] PUT /identity/v3/projects/a10ac2feed034f8c84411379d048d9b9/users/691126a73424498ea86d1dcdd8ffe371/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:25:41.599385 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d9790a-9636-40d7-bfe9-a21347c84349 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:41.599644 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d9790a-9636-40d7-bfe9-a21347c84349 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:41.599924 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d9790a-9636-40d7-bfe9-a21347c84349 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:41.647515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 898/1789] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:41 2022] PUT /identity/v3/projects/e7cc49c6d4ea4cae924c6af52494e6f0/users/a5a5f29fc50e4fdfb8a9fa6fbac48099/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:25:41.650763 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f6d9790a-9636-40d7-bfe9-a21347c84349 None None] MFA Rules not processed for user `691126a73424498ea86d1dcdd8ffe371`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:41.652309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f6d9790a-9636-40d7-bfe9-a21347c84349 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:41.659487 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf7f4db-c716-4028-8950-4028e9fe3d34 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:41.660338 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf7f4db-c716-4028-8950-4028e9fe3d34 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:41.661805 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf7f4db-c716-4028-8950-4028e9fe3d34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:41.709781 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-bdf7f4db-c716-4028-8950-4028e9fe3d34 None None] MFA Rules not processed for user `a5a5f29fc50e4fdfb8a9fa6fbac48099`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:41.711789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-bdf7f4db-c716-4028-8950-4028e9fe3d34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:41.755827 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 892/1790] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:41 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:25:41.760864 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d9790a-9636-40d7-bfe9-a21347c84349 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:41.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 899/1791] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:41 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:25:41.808829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf7f4db-c716-4028-8950-4028e9fe3d34 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:41.833359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d9790a-9636-40d7-bfe9-a21347c84349 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:41.836867 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16df5ce-fb77-48fb-9c0c-ebaa3323a3f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:41.837472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16df5ce-fb77-48fb-9c0c-ebaa3323a3f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:41.837747 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16df5ce-fb77-48fb-9c0c-ebaa3323a3f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:41.838012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16df5ce-fb77-48fb-9c0c-ebaa3323a3f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:41.838623 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16df5ce-fb77-48fb-9c0c-ebaa3323a3f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:41.840580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16df5ce-fb77-48fb-9c0c-ebaa3323a3f9 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:41.884004 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf7f4db-c716-4028-8950-4028e9fe3d34 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:41.888873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef469ddc-5216-4ffa-bfcd-b1b9e5c7d13e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:41.889534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef469ddc-5216-4ffa-bfcd-b1b9e5c7d13e admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:41.889808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef469ddc-5216-4ffa-bfcd-b1b9e5c7d13e admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:41.890073 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef469ddc-5216-4ffa-bfcd-b1b9e5c7d13e admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:41.890694 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef469ddc-5216-4ffa-bfcd-b1b9e5c7d13e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:41.892748 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef469ddc-5216-4ffa-bfcd-b1b9e5c7d13e admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:41.957318 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 893/1792] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:41 2022] POST /identity/v3/projects => generated 377 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:41.961574 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16df5ce-fb77-48fb-9c0c-ebaa3323a3f9 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:41.982644 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 900/1793] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:41 2022] POST /identity/v3/projects => generated 387 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:25:41.986977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef469ddc-5216-4ffa-bfcd-b1b9e5c7d13e admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:42.031814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16df5ce-fb77-48fb-9c0c-ebaa3323a3f9 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:42.068389 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef469ddc-5216-4ffa-bfcd-b1b9e5c7d13e admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:42.074082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af969b82-bc31-4ef3-a72a-3a708f179f2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:42.074697 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af969b82-bc31-4ef3-a72a-3a708f179f2f admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:42.074972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af969b82-bc31-4ef3-a72a-3a708f179f2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:42.075241 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af969b82-bc31-4ef3-a72a-3a708f179f2f admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:42.075863 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af969b82-bc31-4ef3-a72a-3a708f179f2f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:42.078006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af969b82-bc31-4ef3-a72a-3a708f179f2f admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:42.095864 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1fa95a-0b64-4a49-82d4-166355075b5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:42.096531 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1fa95a-0b64-4a49-82d4-166355075b5b admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:42.096833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1fa95a-0b64-4a49-82d4-166355075b5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:42.097106 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1fa95a-0b64-4a49-82d4-166355075b5b admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:42.097727 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1fa95a-0b64-4a49-82d4-166355075b5b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:42.099796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1fa95a-0b64-4a49-82d4-166355075b5b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:42.235575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 901/1794] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:41 2022] POST /identity/v3/users => generated 300 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:25:42.236620 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af969b82-bc31-4ef3-a72a-3a708f179f2f admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:42.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af969b82-bc31-4ef3-a72a-3a708f179f2f admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:42.252547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168996be-bbe6-470d-bb84-3bad2c93f0c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:42.253413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168996be-bbe6-470d-bb84-3bad2c93f0c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:42.253413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168996be-bbe6-470d-bb84-3bad2c93f0c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:42.254151 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168996be-bbe6-470d-bb84-3bad2c93f0c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:42.255076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168996be-bbe6-470d-bb84-3bad2c93f0c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:42.257075 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 894/1795] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:41 2022] POST /identity/v3/users => generated 295 bytes in 294 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:25:42.257392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168996be-bbe6-470d-bb84-3bad2c93f0c0 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:42.260044 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1fa95a-0b64-4a49-82d4-166355075b5b admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:42.272442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 902/1796] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:42 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:42.273669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168996be-bbe6-470d-bb84-3bad2c93f0c0 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:42.274539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1fa95a-0b64-4a49-82d4-166355075b5b admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:42.281364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03a3a04-84e9-4e91-8c0a-a5dc5699a56a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:42.281902 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03a3a04-84e9-4e91-8c0a-a5dc5699a56a admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:42.284125 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168996be-bbe6-470d-bb84-3bad2c93f0c0 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:42.284267 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03a3a04-84e9-4e91-8c0a-a5dc5699a56a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:42.284549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03a3a04-84e9-4e91-8c0a-a5dc5699a56a admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:42.285534 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03a3a04-84e9-4e91-8c0a-a5dc5699a56a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:42.287467 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03a3a04-84e9-4e91-8c0a-a5dc5699a56a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:42.290956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556bfbd1-00af-43bd-9979-399a02af28f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:42.294255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556bfbd1-00af-43bd-9979-399a02af28f8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:42.294255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556bfbd1-00af-43bd-9979-399a02af28f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:42.294255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556bfbd1-00af-43bd-9979-399a02af28f8 admin admin] PATH_INFO: `/v3/projects/93903773f3114f1f95903be22f5a7de3/users/078016b80add405eab8d2d80dafdc913/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:42.303282 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 895/1797] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:42 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:25:42.304216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03a3a04-84e9-4e91-8c0a-a5dc5699a56a admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:42.320094 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03a3a04-84e9-4e91-8c0a-a5dc5699a56a admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:42.325275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcb6740-6603-4bd9-a1e5-e5eea59d82eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:42.325940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcb6740-6603-4bd9-a1e5-e5eea59d82eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:42.326220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcb6740-6603-4bd9-a1e5-e5eea59d82eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:42.326491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcb6740-6603-4bd9-a1e5-e5eea59d82eb admin admin] PATH_INFO: `/v3/projects/c41a7542eda74d04abaab1998077a8d5/users/67b83f40a81b477eb0eab6a5026054eb/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:42.333588 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556bfbd1-00af-43bd-9979-399a02af28f8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=93903773f3114f1f95903be22f5a7de3, user_id=078016b80add405eab8d2d80dafdc913, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:42.337876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556bfbd1-00af-43bd-9979-399a02af28f8 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:42.349337 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcb6740-6603-4bd9-a1e5-e5eea59d82eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=c41a7542eda74d04abaab1998077a8d5, user_id=67b83f40a81b477eb0eab6a5026054eb, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:42.353692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcb6740-6603-4bd9-a1e5-e5eea59d82eb admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:42.434240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 903/1798] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:42 2022] PUT /identity/v3/projects/93903773f3114f1f95903be22f5a7de3/users/078016b80add405eab8d2d80dafdc913/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:42.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556bfbd1-00af-43bd-9979-399a02af28f8 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:42.455402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 896/1799] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:42 2022] PUT /identity/v3/projects/c41a7542eda74d04abaab1998077a8d5/users/67b83f40a81b477eb0eab6a5026054eb/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:42.459163 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcb6740-6603-4bd9-a1e5-e5eea59d82eb admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:42.507360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556bfbd1-00af-43bd-9979-399a02af28f8 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:42.525168 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcb6740-6603-4bd9-a1e5-e5eea59d82eb admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:42.530191 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd169f7c-bc61-4eb1-bd9a-74387078d5c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:42.530799 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd169f7c-bc61-4eb1-bd9a-74387078d5c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:42.531079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd169f7c-bc61-4eb1-bd9a-74387078d5c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:42.531350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd169f7c-bc61-4eb1-bd9a-74387078d5c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:42.532094 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd169f7c-bc61-4eb1-bd9a-74387078d5c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:42.533779 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd169f7c-bc61-4eb1-bd9a-74387078d5c6 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:42.544675 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 897/1800] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:42 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:42.548769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd169f7c-bc61-4eb1-bd9a-74387078d5c6 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:42.559470 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd169f7c-bc61-4eb1-bd9a-74387078d5c6 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:42.565240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6102b4a5-74db-440e-a2e2-590c007618d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:42.565867 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6102b4a5-74db-440e-a2e2-590c007618d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:42.566101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6102b4a5-74db-440e-a2e2-590c007618d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:42.566373 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6102b4a5-74db-440e-a2e2-590c007618d7 admin admin] PATH_INFO: `/v3/projects/c41a7542eda74d04abaab1998077a8d5/users/67b83f40a81b477eb0eab6a5026054eb/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:42.568413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6102b4a5-74db-440e-a2e2-590c007618d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c41a7542eda74d04abaab1998077a8d5, user_id=67b83f40a81b477eb0eab6a5026054eb, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:42.572446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1a1ef2-08a4-478a-9bfe-d41f9751ce7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:42.572567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6102b4a5-74db-440e-a2e2-590c007618d7 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:42.572667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1a1ef2-08a4-478a-9bfe-d41f9751ce7b admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:42.572667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1a1ef2-08a4-478a-9bfe-d41f9751ce7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:42.572667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1a1ef2-08a4-478a-9bfe-d41f9751ce7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:42.572667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d1a1ef2-08a4-478a-9bfe-d41f9751ce7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:42.574293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d1a1ef2-08a4-478a-9bfe-d41f9751ce7b admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:42.586160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 904/1801] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:42 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:25:42.590201 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1a1ef2-08a4-478a-9bfe-d41f9751ce7b admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:42.591153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 898/1802] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:42 2022] PUT /identity/v3/projects/c41a7542eda74d04abaab1998077a8d5/users/67b83f40a81b477eb0eab6a5026054eb/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:25:42.601931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1a1ef2-08a4-478a-9bfe-d41f9751ce7b admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:42.602106 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2efd1e-0ec2-4c41-84af-c2b5e9f8f3bb None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:42.602300 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2efd1e-0ec2-4c41-84af-c2b5e9f8f3bb None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:42.602577 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2efd1e-0ec2-4c41-84af-c2b5e9f8f3bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:42.630809 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-bd2efd1e-0ec2-4c41-84af-c2b5e9f8f3bb None None] MFA Rules not processed for user `67b83f40a81b477eb0eab6a5026054eb`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:42.632035 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-bd2efd1e-0ec2-4c41-84af-c2b5e9f8f3bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:42.659540 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de4ae88-16c2-4702-bde5-c0e8228ca66b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:42.660306 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de4ae88-16c2-4702-bde5-c0e8228ca66b admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:42.660585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de4ae88-16c2-4702-bde5-c0e8228ca66b admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:42.660963 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de4ae88-16c2-4702-bde5-c0e8228ca66b admin admin] PATH_INFO: `/v3/projects/93903773f3114f1f95903be22f5a7de3/users/078016b80add405eab8d2d80dafdc913/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:42.663970 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de4ae88-16c2-4702-bde5-c0e8228ca66b admin admin] RBAC: Authorizing `identity:create_grant(project_id=93903773f3114f1f95903be22f5a7de3, user_id=078016b80add405eab8d2d80dafdc913, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:42.666745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de4ae88-16c2-4702-bde5-c0e8228ca66b admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:42.677078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 905/1803] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:42 2022] PUT /identity/v3/projects/93903773f3114f1f95903be22f5a7de3/users/078016b80add405eab8d2d80dafdc913/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:25:42.687186 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aee74c6-a23f-449e-ae02-15160a4ede93 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:42.687406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aee74c6-a23f-449e-ae02-15160a4ede93 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:42.687674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aee74c6-a23f-449e-ae02-15160a4ede93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:42.705733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 899/1804] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:42 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:25:42.715650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-1aee74c6-a23f-449e-ae02-15160a4ede93 None None] MFA Rules not processed for user `078016b80add405eab8d2d80dafdc913`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:42.717234 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1aee74c6-a23f-449e-ae02-15160a4ede93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:42.718324 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2efd1e-0ec2-4c41-84af-c2b5e9f8f3bb None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:42.719664 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-bd2efd1e-0ec2-4c41-84af-c2b5e9f8f3bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:42.777259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2efd1e-0ec2-4c41-84af-c2b5e9f8f3bb None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:42.788191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 906/1805] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:42 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:25:42.800933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aee74c6-a23f-449e-ae02-15160a4ede93 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:42.801478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1aee74c6-a23f-449e-ae02-15160a4ede93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:42.826996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac454f9-a90f-4351-9b28-80bc37f3cf2a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:42.827611 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac454f9-a90f-4351-9b28-80bc37f3cf2a service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:42.829638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac454f9-a90f-4351-9b28-80bc37f3cf2a service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:42.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac454f9-a90f-4351-9b28-80bc37f3cf2a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:42.861101 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aee74c6-a23f-449e-ae02-15160a4ede93 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:42.865692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36208b24-e29b-48db-a41c-133c808795f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:42.866589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36208b24-e29b-48db-a41c-133c808795f9 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:42.866924 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36208b24-e29b-48db-a41c-133c808795f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:42.867206 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36208b24-e29b-48db-a41c-133c808795f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:42.877223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36208b24-e29b-48db-a41c-133c808795f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:42.881111 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36208b24-e29b-48db-a41c-133c808795f9 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:42.889736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 907/1806] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:42 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:25:42.899455 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac454f9-a90f-4351-9b28-80bc37f3cf2a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:42.904829 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac454f9-a90f-4351-9b28-80bc37f3cf2a service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:42.913351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 900/1807] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:42 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:42.939393 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36208b24-e29b-48db-a41c-133c808795f9 service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:43.018367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36208b24-e29b-48db-a41c-133c808795f9 service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:43.023686 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4069bdc-c9df-4f2c-8ad1-21f5936908b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:43.024346 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4069bdc-c9df-4f2c-8ad1-21f5936908b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:43.024621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4069bdc-c9df-4f2c-8ad1-21f5936908b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:43.024890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4069bdc-c9df-4f2c-8ad1-21f5936908b5 admin admin] PATH_INFO: `/v3/users/a5a5f29fc50e4fdfb8a9fa6fbac48099` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:43.026014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4069bdc-c9df-4f2c-8ad1-21f5936908b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5a5f29fc50e4fdfb8a9fa6fbac48099)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:43.027916 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4069bdc-c9df-4f2c-8ad1-21f5936908b5 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:43.207514 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e4069bdc-c9df-4f2c-8ad1-21f5936908b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5a5f29fc50e4fdfb8a9fa6fbac48099'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:43.217291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e4069bdc-c9df-4f2c-8ad1-21f5936908b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5a5f29fc50e4fdfb8a9fa6fbac48099'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:43.234143 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e4069bdc-c9df-4f2c-8ad1-21f5936908b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5a5f29fc50e4fdfb8a9fa6fbac48099'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:43.315753 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e4069bdc-c9df-4f2c-8ad1-21f5936908b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5a5f29fc50e4fdfb8a9fa6fbac48099'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:43.316439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e4069bdc-c9df-4f2c-8ad1-21f5936908b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5a5f29fc50e4fdfb8a9fa6fbac48099'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:43.328589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 908/1808] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:42 2022] DELETE /identity/v3/users/a5a5f29fc50e4fdfb8a9fa6fbac48099 => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:43.331786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be14f70-dc5d-4075-8ca9-e5bdced0ab52 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:43.332183 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be14f70-dc5d-4075-8ca9-e5bdced0ab52 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:43.332518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be14f70-dc5d-4075-8ca9-e5bdced0ab52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:43.370011 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-6be14f70-dc5d-4075-8ca9-e5bdced0ab52 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:43.371517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-6be14f70-dc5d-4075-8ca9-e5bdced0ab52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:43.481686 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:43.501724 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 901/1809] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:43.596016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4069bdc-c9df-4f2c-8ad1-21f5936908b5 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:43.616646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e4069bdc-c9df-4f2c-8ad1-21f5936908b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:43.673479 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:43.766455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4069bdc-c9df-4f2c-8ad1-21f5936908b5 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:43.772267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa7f863-c2b8-43a3-9b9c-49fd51dfc067 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:43.772963 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa7f863-c2b8-43a3-9b9c-49fd51dfc067 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:43.773243 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa7f863-c2b8-43a3-9b9c-49fd51dfc067 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:43.773650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa7f863-c2b8-43a3-9b9c-49fd51dfc067 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:43.785637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa7f863-c2b8-43a3-9b9c-49fd51dfc067 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:43.790132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa7f863-c2b8-43a3-9b9c-49fd51dfc067 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:43.817263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be14f70-dc5d-4075-8ca9-e5bdced0ab52 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:43.818797 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-6be14f70-dc5d-4075-8ca9-e5bdced0ab52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:43.836514 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 909/1810] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 243 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:43.863070 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be14f70-dc5d-4075-8ca9-e5bdced0ab52 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:43.863808 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0172d8-36de-430b-a93b-56fd1c6ed757 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:43.865341 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0172d8-36de-430b-a93b-56fd1c6ed757 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:43.867007 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0172d8-36de-430b-a93b-56fd1c6ed757 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:43.867795 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0172d8-36de-430b-a93b-56fd1c6ed757 admin admin] PATH_INFO: `/v3/users/691126a73424498ea86d1dcdd8ffe371` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:43.870134 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0172d8-36de-430b-a93b-56fd1c6ed757 admin admin] RBAC: Authorizing `identity:delete_user(user_id=691126a73424498ea86d1dcdd8ffe371)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:43.872573 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0172d8-36de-430b-a93b-56fd1c6ed757 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:44.327640 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1a0172d8-36de-430b-a93b-56fd1c6ed757 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '691126a73424498ea86d1dcdd8ffe371'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:44.346503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1a0172d8-36de-430b-a93b-56fd1c6ed757 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '691126a73424498ea86d1dcdd8ffe371'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:44.348271 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1a0172d8-36de-430b-a93b-56fd1c6ed757 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '691126a73424498ea86d1dcdd8ffe371'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:44.405336 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1a0172d8-36de-430b-a93b-56fd1c6ed757 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '691126a73424498ea86d1dcdd8ffe371'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:44.552937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1a0172d8-36de-430b-a93b-56fd1c6ed757 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '691126a73424498ea86d1dcdd8ffe371'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:44.566725 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 902/1811] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:43 2022] DELETE /identity/v3/users/691126a73424498ea86d1dcdd8ffe371 => generated 0 bytes in 743 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:25:44.566883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8fb551-e6e1-4b0b-9f88-37e73d318aed None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:44.566883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8fb551-e6e1-4b0b-9f88-37e73d318aed None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:44.566883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8fb551-e6e1-4b0b-9f88-37e73d318aed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:44.605327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-0b8fb551-e6e1-4b0b-9f88-37e73d318aed None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:44.606561 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-0b8fb551-e6e1-4b0b-9f88-37e73d318aed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:44.681931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 910/1812] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:44.693989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0172d8-36de-430b-a93b-56fd1c6ed757 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:44.696994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-1a0172d8-36de-430b-a93b-56fd1c6ed757 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:44.768346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0172d8-36de-430b-a93b-56fd1c6ed757 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:44.773124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72cf434-dfad-4ee6-9708-b28565e7e9b4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:44.773777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72cf434-dfad-4ee6-9708-b28565e7e9b4 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:44.774105 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72cf434-dfad-4ee6-9708-b28565e7e9b4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:44.774382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72cf434-dfad-4ee6-9708-b28565e7e9b4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:44.793339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72cf434-dfad-4ee6-9708-b28565e7e9b4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:44.795349 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72cf434-dfad-4ee6-9708-b28565e7e9b4 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:44.811431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 903/1813] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:45.233481 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8fb551-e6e1-4b0b-9f88-37e73d318aed None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:45.235597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-0b8fb551-e6e1-4b0b-9f88-37e73d318aed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:45.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8fb551-e6e1-4b0b-9f88-37e73d318aed None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:45.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678120e0-2532-4d0a-b89d-6dd3b00b71ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:45.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678120e0-2532-4d0a-b89d-6dd3b00b71ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:45.265565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678120e0-2532-4d0a-b89d-6dd3b00b71ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:45.266387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678120e0-2532-4d0a-b89d-6dd3b00b71ea admin admin] PATH_INFO: `/v3/projects/e7cc49c6d4ea4cae924c6af52494e6f0` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:45.266387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678120e0-2532-4d0a-b89d-6dd3b00b71ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=e7cc49c6d4ea4cae924c6af52494e6f0)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:45.266387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678120e0-2532-4d0a-b89d-6dd3b00b71ea admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:45.374992 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72cf434-dfad-4ee6-9708-b28565e7e9b4 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:45.376582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-c72cf434-dfad-4ee6-9708-b28565e7e9b4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:45.412628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72cf434-dfad-4ee6-9708-b28565e7e9b4 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:45.451085 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bc28da-10b9-470b-89a7-645a70d83725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:45.454655 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bc28da-10b9-470b-89a7-645a70d83725 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:45.458573 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bc28da-10b9-470b-89a7-645a70d83725 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:45.459536 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bc28da-10b9-470b-89a7-645a70d83725 admin admin] PATH_INFO: `/v3/projects/a10ac2feed034f8c84411379d048d9b9` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:45.469212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bc28da-10b9-470b-89a7-645a70d83725 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a10ac2feed034f8c84411379d048d9b9)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:45.474230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bc28da-10b9-470b-89a7-645a70d83725 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:45.477073 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-678120e0-2532-4d0a-b89d-6dd3b00b71ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e7cc49c6d4ea4cae924c6af52494e6f0', 'request_id': 'req-678120e0-2532-4d0a-b89d-6dd3b00b71ea'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:45.591291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-678120e0-2532-4d0a-b89d-6dd3b00b71ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e7cc49c6d4ea4cae924c6af52494e6f0', 'request_id': 'req-678120e0-2532-4d0a-b89d-6dd3b00b71ea'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:45.606831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-678120e0-2532-4d0a-b89d-6dd3b00b71ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e7cc49c6d4ea4cae924c6af52494e6f0', 'request_id': 'req-678120e0-2532-4d0a-b89d-6dd3b00b71ea'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:45.634840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-678120e0-2532-4d0a-b89d-6dd3b00b71ea admin admin] The token cache is being invalidate because project e7cc49c6d4ea4cae924c6af52494e6f0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:45.635440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-678120e0-2532-4d0a-b89d-6dd3b00b71ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:45.636835 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-678120e0-2532-4d0a-b89d-6dd3b00b71ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:45.640918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 911/1814] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:45 2022] DELETE /identity/v3/projects/e7cc49c6d4ea4cae924c6af52494e6f0 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:25:45.649073 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678120e0-2532-4d0a-b89d-6dd3b00b71ea admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:45.650149 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-678120e0-2532-4d0a-b89d-6dd3b00b71ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:45.667353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-63bc28da-10b9-470b-89a7-645a70d83725 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a10ac2feed034f8c84411379d048d9b9', 'request_id': 'req-63bc28da-10b9-470b-89a7-645a70d83725'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:45.678964 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-63bc28da-10b9-470b-89a7-645a70d83725 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a10ac2feed034f8c84411379d048d9b9', 'request_id': 'req-63bc28da-10b9-470b-89a7-645a70d83725'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:45.690452 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-63bc28da-10b9-470b-89a7-645a70d83725 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a10ac2feed034f8c84411379d048d9b9', 'request_id': 'req-63bc28da-10b9-470b-89a7-645a70d83725'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:45.778246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678120e0-2532-4d0a-b89d-6dd3b00b71ea admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:45.866233 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaaef1ac-7b7f-462e-b3b0-48b8a03dce34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:45.867505 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaaef1ac-7b7f-462e-b3b0-48b8a03dce34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:45.868000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaaef1ac-7b7f-462e-b3b0-48b8a03dce34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:45.868766 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaaef1ac-7b7f-462e-b3b0-48b8a03dce34 admin admin] PATH_INFO: `/v3/users/078016b80add405eab8d2d80dafdc913` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:45.870637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaaef1ac-7b7f-462e-b3b0-48b8a03dce34 admin admin] RBAC: Authorizing `identity:delete_user(user_id=078016b80add405eab8d2d80dafdc913)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:45.872991 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaaef1ac-7b7f-462e-b3b0-48b8a03dce34 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:45.873180 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-63bc28da-10b9-470b-89a7-645a70d83725 admin admin] The token cache is being invalidate because project a10ac2feed034f8c84411379d048d9b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:45.873592 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-63bc28da-10b9-470b-89a7-645a70d83725 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:45.879321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-63bc28da-10b9-470b-89a7-645a70d83725 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:45.888061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 904/1815] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:45 2022] DELETE /identity/v3/projects/a10ac2feed034f8c84411379d048d9b9 => generated 0 bytes in 511 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:45.889482 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bc28da-10b9-470b-89a7-645a70d83725 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:45.892784 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-63bc28da-10b9-470b-89a7-645a70d83725 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:45.918641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bc28da-10b9-470b-89a7-645a70d83725 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:45.928406 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaac3c8d-bfe8-4d1e-be96-3589bf9f47d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:45.929693 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaac3c8d-bfe8-4d1e-be96-3589bf9f47d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:45.930216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaac3c8d-bfe8-4d1e-be96-3589bf9f47d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:45.930676 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaac3c8d-bfe8-4d1e-be96-3589bf9f47d5 admin admin] PATH_INFO: `/v3/users/67b83f40a81b477eb0eab6a5026054eb` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:45.932692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaac3c8d-bfe8-4d1e-be96-3589bf9f47d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=67b83f40a81b477eb0eab6a5026054eb)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:45.935390 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaac3c8d-bfe8-4d1e-be96-3589bf9f47d5 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:46.162984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-eaaef1ac-7b7f-462e-b3b0-48b8a03dce34 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '078016b80add405eab8d2d80dafdc913'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:46.171132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-eaaef1ac-7b7f-462e-b3b0-48b8a03dce34 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '078016b80add405eab8d2d80dafdc913'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:46.204914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-eaaef1ac-7b7f-462e-b3b0-48b8a03dce34 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '078016b80add405eab8d2d80dafdc913'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:46.224737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-aaac3c8d-bfe8-4d1e-be96-3589bf9f47d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '67b83f40a81b477eb0eab6a5026054eb'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:46.251384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-aaac3c8d-bfe8-4d1e-be96-3589bf9f47d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '67b83f40a81b477eb0eab6a5026054eb'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:46.252182 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-aaac3c8d-bfe8-4d1e-be96-3589bf9f47d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '67b83f40a81b477eb0eab6a5026054eb'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:46.273727 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-aaac3c8d-bfe8-4d1e-be96-3589bf9f47d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '67b83f40a81b477eb0eab6a5026054eb'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:46.315554 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-eaaef1ac-7b7f-462e-b3b0-48b8a03dce34 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '078016b80add405eab8d2d80dafdc913'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:46.316611 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-eaaef1ac-7b7f-462e-b3b0-48b8a03dce34 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '078016b80add405eab8d2d80dafdc913'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:46.326635 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 912/1816] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:45 2022] DELETE /identity/v3/users/078016b80add405eab8d2d80dafdc913 => generated 0 bytes in 681 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:46.351212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:46.464837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-aaac3c8d-bfe8-4d1e-be96-3589bf9f47d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '67b83f40a81b477eb0eab6a5026054eb'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:46.482929 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 905/1817] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:45 2022] DELETE /identity/v3/users/67b83f40a81b477eb0eab6a5026054eb => generated 0 bytes in 596 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:46.761784 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:47.642648 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaaef1ac-7b7f-462e-b3b0-48b8a03dce34 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:47.663628 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-eaaef1ac-7b7f-462e-b3b0-48b8a03dce34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:47.794009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaaef1ac-7b7f-462e-b3b0-48b8a03dce34 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:47.884677 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a836a9-772b-49e8-a5f5-7c1a814c1f65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:47.884929 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a836a9-772b-49e8-a5f5-7c1a814c1f65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:47.885233 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a836a9-772b-49e8-a5f5-7c1a814c1f65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:47.885583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a836a9-772b-49e8-a5f5-7c1a814c1f65 admin admin] PATH_INFO: `/v3/projects/c41a7542eda74d04abaab1998077a8d5` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:47.889504 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a836a9-772b-49e8-a5f5-7c1a814c1f65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c41a7542eda74d04abaab1998077a8d5)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:47.892080 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a836a9-772b-49e8-a5f5-7c1a814c1f65 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:47.983371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaac3c8d-bfe8-4d1e-be96-3589bf9f47d5 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:47.986627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-aaac3c8d-bfe8-4d1e-be96-3589bf9f47d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:48.007988 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaac3c8d-bfe8-4d1e-be96-3589bf9f47d5 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:48.032620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31b34cf-bc02-4784-9d1e-4fac13a2251f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:48.033332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31b34cf-bc02-4784-9d1e-4fac13a2251f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:48.033979 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31b34cf-bc02-4784-9d1e-4fac13a2251f admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:48.034737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31b34cf-bc02-4784-9d1e-4fac13a2251f admin admin] PATH_INFO: `/v3/projects/93903773f3114f1f95903be22f5a7de3` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:48.044767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31b34cf-bc02-4784-9d1e-4fac13a2251f admin admin] RBAC: Authorizing `identity:delete_project(project_id=93903773f3114f1f95903be22f5a7de3)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:48.044767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31b34cf-bc02-4784-9d1e-4fac13a2251f admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:48.122933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-67a836a9-772b-49e8-a5f5-7c1a814c1f65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c41a7542eda74d04abaab1998077a8d5', 'request_id': 'req-67a836a9-772b-49e8-a5f5-7c1a814c1f65'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:48.206706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-67a836a9-772b-49e8-a5f5-7c1a814c1f65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c41a7542eda74d04abaab1998077a8d5', 'request_id': 'req-67a836a9-772b-49e8-a5f5-7c1a814c1f65'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:48.213717 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-67a836a9-772b-49e8-a5f5-7c1a814c1f65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c41a7542eda74d04abaab1998077a8d5', 'request_id': 'req-67a836a9-772b-49e8-a5f5-7c1a814c1f65'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:48.229655 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-67a836a9-772b-49e8-a5f5-7c1a814c1f65 admin admin] The token cache is being invalidate because project c41a7542eda74d04abaab1998077a8d5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:48.229655 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-67a836a9-772b-49e8-a5f5-7c1a814c1f65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:48.229655 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-67a836a9-772b-49e8-a5f5-7c1a814c1f65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:48.230195 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e31b34cf-bc02-4784-9d1e-4fac13a2251f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '93903773f3114f1f95903be22f5a7de3', 'request_id': 'req-e31b34cf-bc02-4784-9d1e-4fac13a2251f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:48.230371 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 913/1818] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:47 2022] DELETE /identity/v3/projects/c41a7542eda74d04abaab1998077a8d5 => generated 0 bytes in 589 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:48.233749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d6257c-0f3b-4859-9bef-beec6783d4ac None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:48.234095 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d6257c-0f3b-4859-9bef-beec6783d4ac None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:48.234375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d6257c-0f3b-4859-9bef-beec6783d4ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:48.235859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e31b34cf-bc02-4784-9d1e-4fac13a2251f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '93903773f3114f1f95903be22f5a7de3', 'request_id': 'req-e31b34cf-bc02-4784-9d1e-4fac13a2251f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:48.250223 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e31b34cf-bc02-4784-9d1e-4fac13a2251f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '93903773f3114f1f95903be22f5a7de3', 'request_id': 'req-e31b34cf-bc02-4784-9d1e-4fac13a2251f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:48.257219 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-38d6257c-0f3b-4859-9bef-beec6783d4ac None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:48.258473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-38d6257c-0f3b-4859-9bef-beec6783d4ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:48.325820 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 914/1819] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:25:48.334895 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13be08c3-9511-45f1-a70c-b44963d37b61 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:48.335176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13be08c3-9511-45f1-a70c-b44963d37b61 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:48.335446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13be08c3-9511-45f1-a70c-b44963d37b61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:48.356492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-13be08c3-9511-45f1-a70c-b44963d37b61 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:48.358154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-13be08c3-9511-45f1-a70c-b44963d37b61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:48.367185 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 915/1820] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:48.380073 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13be08c3-9511-45f1-a70c-b44963d37b61 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:48.384326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13be08c3-9511-45f1-a70c-b44963d37b61 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:48.389317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c502ba18-ff91-4486-be80-3b77da46c0a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:48.390140 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c502ba18-ff91-4486-be80-3b77da46c0a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:48.390566 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c502ba18-ff91-4486-be80-3b77da46c0a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:48.390835 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c502ba18-ff91-4486-be80-3b77da46c0a5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:48.391485 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c502ba18-ff91-4486-be80-3b77da46c0a5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:25:48.391781 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c502ba18-ff91-4486-be80-3b77da46c0a5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:48.393442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c502ba18-ff91-4486-be80-3b77da46c0a5 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:48.406817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 916/1821] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:25:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:25:48.416214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e31b34cf-bc02-4784-9d1e-4fac13a2251f admin admin] The token cache is being invalidate because project 93903773f3114f1f95903be22f5a7de3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:48.416214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e31b34cf-bc02-4784-9d1e-4fac13a2251f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:48.416214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e31b34cf-bc02-4784-9d1e-4fac13a2251f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:48.417125 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 906/1822] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:47 2022] DELETE /identity/v3/projects/93903773f3114f1f95903be22f5a7de3 => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:25:48.421195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c502ba18-ff91-4486-be80-3b77da46c0a5 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:48.422248 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a158a61-9884-41e3-81f8-cce13ec3b140 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:48.422248 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a158a61-9884-41e3-81f8-cce13ec3b140 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:48.422480 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a158a61-9884-41e3-81f8-cce13ec3b140 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:48.423981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c502ba18-ff91-4486-be80-3b77da46c0a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:48.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c502ba18-ff91-4486-be80-3b77da46c0a5 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:48.444353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320080ea-7a29-4d90-a565-28e089257a57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:48.444987 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-1a158a61-9884-41e3-81f8-cce13ec3b140 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:48.445113 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320080ea-7a29-4d90-a565-28e089257a57 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:48.445437 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320080ea-7a29-4d90-a565-28e089257a57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:48.445768 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320080ea-7a29-4d90-a565-28e089257a57 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:48.446460 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320080ea-7a29-4d90-a565-28e089257a57 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:48.446582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-1a158a61-9884-41e3-81f8-cce13ec3b140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:48.450861 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320080ea-7a29-4d90-a565-28e089257a57 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:48.455643 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 907/1823] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:48.468769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0597a86f-7742-48ea-8c73-ac861b52906e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:48.468769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0597a86f-7742-48ea-8c73-ac861b52906e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:48.468769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0597a86f-7742-48ea-8c73-ac861b52906e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:48.502344 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-0597a86f-7742-48ea-8c73-ac861b52906e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:48.503577 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-0597a86f-7742-48ea-8c73-ac861b52906e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:48.512700 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 908/1824] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:25:48.517195 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0597a86f-7742-48ea-8c73-ac861b52906e None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:48.527991 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0597a86f-7742-48ea-8c73-ac861b52906e None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:48.536056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f790d22f-178c-46a0-abe3-d4a40fe72571 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:48.536924 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f790d22f-178c-46a0-abe3-d4a40fe72571 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:48.537427 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f790d22f-178c-46a0-abe3-d4a40fe72571 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:48.538013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f790d22f-178c-46a0-abe3-d4a40fe72571 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:48.538969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f790d22f-178c-46a0-abe3-d4a40fe72571 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:25:48.539273 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f790d22f-178c-46a0-abe3-d4a40fe72571 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:48.540924 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f790d22f-178c-46a0-abe3-d4a40fe72571 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:48.559124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 909/1825] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:25:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:25:48.567380 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f790d22f-178c-46a0-abe3-d4a40fe72571 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:48.578987 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f790d22f-178c-46a0-abe3-d4a40fe72571 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:48.586103 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb43845-6a1d-4314-9330-53877007c675 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:48.586740 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb43845-6a1d-4314-9330-53877007c675 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:48.587020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb43845-6a1d-4314-9330-53877007c675 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:48.587538 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb43845-6a1d-4314-9330-53877007c675 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:48.588232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb43845-6a1d-4314-9330-53877007c675 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:48.590384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb43845-6a1d-4314-9330-53877007c675 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:48.597524 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 917/1826] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:48 2022] POST /identity/v3/projects => generated 385 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:48.603026 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320080ea-7a29-4d90-a565-28e089257a57 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:48.690898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320080ea-7a29-4d90-a565-28e089257a57 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:48.696318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559ff7cb-a42a-4a76-af47-d090c84cae84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:48.699076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559ff7cb-a42a-4a76-af47-d090c84cae84 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:48.699372 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559ff7cb-a42a-4a76-af47-d090c84cae84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:48.699683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559ff7cb-a42a-4a76-af47-d090c84cae84 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:48.700331 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559ff7cb-a42a-4a76-af47-d090c84cae84 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:48.702324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559ff7cb-a42a-4a76-af47-d090c84cae84 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:48.757723 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 910/1827] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:48 2022] POST /identity/v3/projects => generated 401 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:25:48.759010 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb43845-6a1d-4314-9330-53877007c675 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:48.862804 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb43845-6a1d-4314-9330-53877007c675 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:48.870242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 918/1828] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:48 2022] POST /identity/v3/users => generated 293 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:48.870242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559ff7cb-a42a-4a76-af47-d090c84cae84 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:48.871672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f7de7d-ce9b-46ef-bcf8-bd398b4cb06b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:48.872372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f7de7d-ce9b-46ef-bcf8-bd398b4cb06b admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:48.872905 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f7de7d-ce9b-46ef-bcf8-bd398b4cb06b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:48.873646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f7de7d-ce9b-46ef-bcf8-bd398b4cb06b admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:48.873822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f7de7d-ce9b-46ef-bcf8-bd398b4cb06b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:48.875789 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f7de7d-ce9b-46ef-bcf8-bd398b4cb06b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:48.884550 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559ff7cb-a42a-4a76-af47-d090c84cae84 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:48.901565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d91dca-13a1-4e1d-97ec-a73445e9b60e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:48.903152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d91dca-13a1-4e1d-97ec-a73445e9b60e admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:48.903541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d91dca-13a1-4e1d-97ec-a73445e9b60e admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:48.903942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d91dca-13a1-4e1d-97ec-a73445e9b60e admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:48.904809 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d91dca-13a1-4e1d-97ec-a73445e9b60e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:48.906718 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d91dca-13a1-4e1d-97ec-a73445e9b60e admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:48.920149 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 919/1829] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:48 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:48.924426 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d91dca-13a1-4e1d-97ec-a73445e9b60e admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:48.933886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:48.943404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d91dca-13a1-4e1d-97ec-a73445e9b60e admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:48.951589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c74d902-e7bc-4f09-8b83-e6a8a69a8e1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:48.952494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c74d902-e7bc-4f09-8b83-e6a8a69a8e1b admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:48.952784 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c74d902-e7bc-4f09-8b83-e6a8a69a8e1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:48.953112 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c74d902-e7bc-4f09-8b83-e6a8a69a8e1b admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:48.953781 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c74d902-e7bc-4f09-8b83-e6a8a69a8e1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:48.955441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c74d902-e7bc-4f09-8b83-e6a8a69a8e1b admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:48.978280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 920/1830] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:48 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:25:48.985097 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c74d902-e7bc-4f09-8b83-e6a8a69a8e1b admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:48.999275 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c74d902-e7bc-4f09-8b83-e6a8a69a8e1b admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:49.005799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3b7b96-cdd7-4c88-8cc0-00954d22687e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:49.006734 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3b7b96-cdd7-4c88-8cc0-00954d22687e admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:49.007107 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3b7b96-cdd7-4c88-8cc0-00954d22687e admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:49.007672 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3b7b96-cdd7-4c88-8cc0-00954d22687e admin admin] PATH_INFO: `/v3/projects/0d062704975147a3b6b674b4e71f8e61/users/f80525e5e3c0428291a95c5e31f8d3bc/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:49.027464 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3b7b96-cdd7-4c88-8cc0-00954d22687e admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d062704975147a3b6b674b4e71f8e61, user_id=f80525e5e3c0428291a95c5e31f8d3bc, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:49.030389 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3b7b96-cdd7-4c88-8cc0-00954d22687e admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:49.203466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 911/1831] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:48 2022] POST /identity/v3/users => generated 301 bytes in 445 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:49.208053 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f7de7d-ce9b-46ef-bcf8-bd398b4cb06b admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:49.226092 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f7de7d-ce9b-46ef-bcf8-bd398b4cb06b admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:49.232981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b4fc05-6242-47c1-8ff5-bc09403d7f32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:49.234431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b4fc05-6242-47c1-8ff5-bc09403d7f32 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:49.236299 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b4fc05-6242-47c1-8ff5-bc09403d7f32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:49.236818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b4fc05-6242-47c1-8ff5-bc09403d7f32 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:49.238319 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b4fc05-6242-47c1-8ff5-bc09403d7f32 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:49.243012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b4fc05-6242-47c1-8ff5-bc09403d7f32 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:49.252284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 921/1832] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:48 2022] PUT /identity/v3/projects/0d062704975147a3b6b674b4e71f8e61/users/f80525e5e3c0428291a95c5e31f8d3bc/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:49.259708 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123ae238-63ff-447d-bc5b-abf1e703bf40 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:49.260383 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123ae238-63ff-447d-bc5b-abf1e703bf40 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:49.260988 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123ae238-63ff-447d-bc5b-abf1e703bf40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:49.274214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 912/1833] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:49 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:49.278389 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b4fc05-6242-47c1-8ff5-bc09403d7f32 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:49.301536 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b4fc05-6242-47c1-8ff5-bc09403d7f32 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:49.347712 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-123ae238-63ff-447d-bc5b-abf1e703bf40 None None] MFA Rules not processed for user `f80525e5e3c0428291a95c5e31f8d3bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:49.349941 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-123ae238-63ff-447d-bc5b-abf1e703bf40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:49.421097 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dcbc9a-3b86-43cb-bd88-a5de53a00122 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:49.422459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dcbc9a-3b86-43cb-bd88-a5de53a00122 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:49.422889 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dcbc9a-3b86-43cb-bd88-a5de53a00122 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:49.423262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dcbc9a-3b86-43cb-bd88-a5de53a00122 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:49.427834 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dcbc9a-3b86-43cb-bd88-a5de53a00122 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:49.431231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dcbc9a-3b86-43cb-bd88-a5de53a00122 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:49.453538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:49.460146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 913/1834] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:49 2022] GET /identity/v3/roles => generated 1420 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:25:49.463993 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dcbc9a-3b86-43cb-bd88-a5de53a00122 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:49.478306 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dcbc9a-3b86-43cb-bd88-a5de53a00122 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:49.491198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47eb6565-aa42-420a-a17e-3a140691dd25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:49.491900 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47eb6565-aa42-420a-a17e-3a140691dd25 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:49.492196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47eb6565-aa42-420a-a17e-3a140691dd25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:49.492473 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47eb6565-aa42-420a-a17e-3a140691dd25 admin admin] PATH_INFO: `/v3/projects/0256ec2f06b249cfb4ab2308adaa8713/users/1064255db4414ac0b28481508ebb4d45/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:49.496760 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 922/1835] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:49 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:25:49.510248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123ae238-63ff-447d-bc5b-abf1e703bf40 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:49.511768 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-123ae238-63ff-447d-bc5b-abf1e703bf40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:49.553412 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47eb6565-aa42-420a-a17e-3a140691dd25 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0256ec2f06b249cfb4ab2308adaa8713, user_id=1064255db4414ac0b28481508ebb4d45, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:49.561738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47eb6565-aa42-420a-a17e-3a140691dd25 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:49.567721 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123ae238-63ff-447d-bc5b-abf1e703bf40 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:49.577288 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049fcb07-5e98-492c-a5a3-1f66a9905ecb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:49.578647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049fcb07-5e98-492c-a5a3-1f66a9905ecb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:49.579057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049fcb07-5e98-492c-a5a3-1f66a9905ecb admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:49.579255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049fcb07-5e98-492c-a5a3-1f66a9905ecb admin admin] PATH_INFO: `/v3/users/e9b2b58261654b67a56e2d0a14adf972` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:49.580567 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049fcb07-5e98-492c-a5a3-1f66a9905ecb admin admin] RBAC: Authorizing `identity:delete_user(user_id=e9b2b58261654b67a56e2d0a14adf972)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:49.583061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049fcb07-5e98-492c-a5a3-1f66a9905ecb admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:49.685168 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 914/1836] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:49 2022] PUT /identity/v3/projects/0256ec2f06b249cfb4ab2308adaa8713/users/1064255db4414ac0b28481508ebb4d45/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:25:49.693375 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47eb6565-aa42-420a-a17e-3a140691dd25 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:49.695359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-47eb6565-aa42-420a-a17e-3a140691dd25 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:49.781429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47eb6565-aa42-420a-a17e-3a140691dd25 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:49.786826 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f414c718-23c7-4c71-a59e-e7cfd2d43511 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:49.789615 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f414c718-23c7-4c71-a59e-e7cfd2d43511 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:49.789615 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f414c718-23c7-4c71-a59e-e7cfd2d43511 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:49.789615 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f414c718-23c7-4c71-a59e-e7cfd2d43511 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:49.812965 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-049fcb07-5e98-492c-a5a3-1f66a9905ecb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e9b2b58261654b67a56e2d0a14adf972'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:49.822197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-049fcb07-5e98-492c-a5a3-1f66a9905ecb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e9b2b58261654b67a56e2d0a14adf972'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:49.845979 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-049fcb07-5e98-492c-a5a3-1f66a9905ecb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e9b2b58261654b67a56e2d0a14adf972'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:49.885121 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f414c718-23c7-4c71-a59e-e7cfd2d43511 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:49.886617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f414c718-23c7-4c71-a59e-e7cfd2d43511 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:49.898220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 915/1837] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:49 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:49.903405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c28ad8-ecdc-4b13-9935-a69760880976 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:49.903405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c28ad8-ecdc-4b13-9935-a69760880976 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:49.903405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c28ad8-ecdc-4b13-9935-a69760880976 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:49.903405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 916/1838] 10.222.0.47 () {54 vars in 921 bytes} [Sun Aug 21 01:25:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:25:49.903405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5770ec2-5952-406d-9457-746a0b402670 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:49.903405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5770ec2-5952-406d-9457-746a0b402670 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:49.906645 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5770ec2-5952-406d-9457-746a0b402670 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:49.926678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-049fcb07-5e98-492c-a5a3-1f66a9905ecb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e9b2b58261654b67a56e2d0a14adf972'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:49.928033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-049fcb07-5e98-492c-a5a3-1f66a9905ecb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e9b2b58261654b67a56e2d0a14adf972'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:49.937289 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 923/1839] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:49 2022] DELETE /identity/v3/users/e9b2b58261654b67a56e2d0a14adf972 => generated 0 bytes in 428 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:25:49.938583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857b09cd-1bec-4857-97e6-088395cc869b None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:49.939879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857b09cd-1bec-4857-97e6-088395cc869b None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:49.940184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857b09cd-1bec-4857-97e6-088395cc869b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:49.952711 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-d5770ec2-5952-406d-9457-746a0b402670 None None] MFA Rules not processed for user `1064255db4414ac0b28481508ebb4d45`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:49.954296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d5770ec2-5952-406d-9457-746a0b402670 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:49.990610 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-857b09cd-1bec-4857-97e6-088395cc869b None None] MFA Rules not processed for user `fabdcdc25d1d4ef6bc66bd6cadecaaf8`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:49.994283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-857b09cd-1bec-4857-97e6-088395cc869b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:50.053612 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 917/1840] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:49 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:50.055054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740ceaa5-792d-447d-9bf0-ccf688686c7f None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:50.055399 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740ceaa5-792d-447d-9bf0-ccf688686c7f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:50.055727 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740ceaa5-792d-447d-9bf0-ccf688686c7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:50.077423 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-740ceaa5-792d-447d-9bf0-ccf688686c7f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:50.078957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-740ceaa5-792d-447d-9bf0-ccf688686c7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:50.149230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 924/1841] 10.222.0.47 () {58 vars in 1018 bytes} [Sun Aug 21 01:25:49 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:50.163523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857b09cd-1bec-4857-97e6-088395cc869b None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:50.166649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-857b09cd-1bec-4857-97e6-088395cc869b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:50.176809 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 918/1842] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:50.177676 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740ceaa5-792d-447d-9bf0-ccf688686c7f None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:50.179417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-740ceaa5-792d-447d-9bf0-ccf688686c7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:50.204366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740ceaa5-792d-447d-9bf0-ccf688686c7f None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:50.211159 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8599e2e-89bd-4962-8244-a48efe556bf3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:50.212026 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8599e2e-89bd-4962-8244-a48efe556bf3 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:50.212353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8599e2e-89bd-4962-8244-a48efe556bf3 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:50.212671 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8599e2e-89bd-4962-8244-a48efe556bf3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:50.227437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8599e2e-89bd-4962-8244-a48efe556bf3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:50.229283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8599e2e-89bd-4962-8244-a48efe556bf3 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:50.241424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 919/1843] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:50.244525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8599e2e-89bd-4962-8244-a48efe556bf3 service nova] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:50.244525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e8599e2e-89bd-4962-8244-a48efe556bf3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:50.260806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857b09cd-1bec-4857-97e6-088395cc869b None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:50.273453 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67787185-90bd-40ce-a2a3-6832c9c5da98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:50.274088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67787185-90bd-40ce-a2a3-6832c9c5da98 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:50.274422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67787185-90bd-40ce-a2a3-6832c9c5da98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:50.274842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67787185-90bd-40ce-a2a3-6832c9c5da98 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:50.275932 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67787185-90bd-40ce-a2a3-6832c9c5da98 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:50.278823 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67787185-90bd-40ce-a2a3-6832c9c5da98 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:50.344140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8599e2e-89bd-4962-8244-a48efe556bf3 service nova] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:50.353278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa477e04-de2c-4faf-8371-81e029c2bbd0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:50.353278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa477e04-de2c-4faf-8371-81e029c2bbd0 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:50.353278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa477e04-de2c-4faf-8371-81e029c2bbd0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:50.353278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa477e04-de2c-4faf-8371-81e029c2bbd0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:50.362193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa477e04-de2c-4faf-8371-81e029c2bbd0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:50.363934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa477e04-de2c-4faf-8371-81e029c2bbd0 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:50.387652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 920/1844] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:50.524207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 925/1845] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:50 2022] POST /identity/v3/projects => generated 401 bytes in 371 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:50.531094 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa477e04-de2c-4faf-8371-81e029c2bbd0 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:50.535652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa477e04-de2c-4faf-8371-81e029c2bbd0 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:50.627971 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6127af63-54f0-4f48-8ed0-9908a12650f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:50.628653 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6127af63-54f0-4f48-8ed0-9908a12650f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:50.629417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6127af63-54f0-4f48-8ed0-9908a12650f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:50.630217 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6127af63-54f0-4f48-8ed0-9908a12650f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:50.630389 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6127af63-54f0-4f48-8ed0-9908a12650f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:50.632368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6127af63-54f0-4f48-8ed0-9908a12650f5 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:50.845452 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 921/1846] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:50 2022] POST /identity/v3/users => generated 307 bytes in 320 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:50.849848 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67787185-90bd-40ce-a2a3-6832c9c5da98 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:50.862513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67787185-90bd-40ce-a2a3-6832c9c5da98 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:50.868729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7714eb-c4cb-42b7-b4d6-f3887bff9d29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:50.869628 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7714eb-c4cb-42b7-b4d6-f3887bff9d29 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:50.870737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7714eb-c4cb-42b7-b4d6-f3887bff9d29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:50.870737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7714eb-c4cb-42b7-b4d6-f3887bff9d29 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:50.871115 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7714eb-c4cb-42b7-b4d6-f3887bff9d29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:50.873109 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7714eb-c4cb-42b7-b4d6-f3887bff9d29 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:50.893798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 926/1847] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:50 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:50.897988 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6127af63-54f0-4f48-8ed0-9908a12650f5 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:50.911705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6127af63-54f0-4f48-8ed0-9908a12650f5 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:50.917504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599761de-13d9-43ae-bc5a-e96b5c194375 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:50.918284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599761de-13d9-43ae-bc5a-e96b5c194375 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:50.918589 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599761de-13d9-43ae-bc5a-e96b5c194375 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:50.918890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599761de-13d9-43ae-bc5a-e96b5c194375 admin admin] PATH_INFO: `/v3/projects/c25040ec09c64334a0016383be054b31/users/427969920afc46ee8adef66c927073be/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:50.939440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599761de-13d9-43ae-bc5a-e96b5c194375 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c25040ec09c64334a0016383be054b31, user_id=427969920afc46ee8adef66c927073be, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:50.944195 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599761de-13d9-43ae-bc5a-e96b5c194375 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:51.108379 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7714eb-c4cb-42b7-b4d6-f3887bff9d29 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:51.110770 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-8e7714eb-c4cb-42b7-b4d6-f3887bff9d29 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:51.128457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7714eb-c4cb-42b7-b4d6-f3887bff9d29 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:51.195665 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabafe88-c610-4023-854b-ee34a7afe439 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:51.195665 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabafe88-c610-4023-854b-ee34a7afe439 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:51.195665 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabafe88-c610-4023-854b-ee34a7afe439 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:51.195665 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabafe88-c610-4023-854b-ee34a7afe439 admin admin] PATH_INFO: `/v3/projects/a02859127960469585afd76d1bf8f6a1` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:51.197245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cabafe88-c610-4023-854b-ee34a7afe439 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a02859127960469585afd76d1bf8f6a1)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:51.197245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cabafe88-c610-4023-854b-ee34a7afe439 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:51.249958 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 922/1848] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:50 2022] PUT /identity/v3/projects/c25040ec09c64334a0016383be054b31/users/427969920afc46ee8adef66c927073be/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:51.254238 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599761de-13d9-43ae-bc5a-e96b5c194375 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:51.345461 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599761de-13d9-43ae-bc5a-e96b5c194375 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:51.352286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2291af06-ab1b-4bf2-9dfd-01d36b90cdd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:51.353022 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2291af06-ab1b-4bf2-9dfd-01d36b90cdd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:51.353487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2291af06-ab1b-4bf2-9dfd-01d36b90cdd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:51.353838 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2291af06-ab1b-4bf2-9dfd-01d36b90cdd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:51.354577 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2291af06-ab1b-4bf2-9dfd-01d36b90cdd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:51.356807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2291af06-ab1b-4bf2-9dfd-01d36b90cdd6 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:51.372306 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 923/1849] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:51 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:25:51.376496 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2291af06-ab1b-4bf2-9dfd-01d36b90cdd6 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:51.388882 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2291af06-ab1b-4bf2-9dfd-01d36b90cdd6 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:51.395826 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44c45ee-86c5-4b0f-9ff0-35682754db61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:51.396757 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44c45ee-86c5-4b0f-9ff0-35682754db61 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:51.397046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44c45ee-86c5-4b0f-9ff0-35682754db61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:51.397510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44c45ee-86c5-4b0f-9ff0-35682754db61 admin admin] PATH_INFO: `/v3/projects/c25040ec09c64334a0016383be054b31/users/427969920afc46ee8adef66c927073be/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:51.399543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44c45ee-86c5-4b0f-9ff0-35682754db61 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c25040ec09c64334a0016383be054b31, user_id=427969920afc46ee8adef66c927073be, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:51.404568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44c45ee-86c5-4b0f-9ff0-35682754db61 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:51.448016 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 924/1850] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:51 2022] PUT /identity/v3/projects/c25040ec09c64334a0016383be054b31/users/427969920afc46ee8adef66c927073be/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:25:51.465239 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30994937-534d-4935-8256-532404d90c0f None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:51.465524 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30994937-534d-4935-8256-532404d90c0f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:51.465803 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30994937-534d-4935-8256-532404d90c0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:51.529207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-30994937-534d-4935-8256-532404d90c0f None None] MFA Rules not processed for user `427969920afc46ee8adef66c927073be`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:51.530870 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-30994937-534d-4935-8256-532404d90c0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:51.535219 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-cabafe88-c610-4023-854b-ee34a7afe439 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a02859127960469585afd76d1bf8f6a1', 'request_id': 'req-cabafe88-c610-4023-854b-ee34a7afe439'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:51.699360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 925/1851] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:51 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:25:51.710999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30994937-534d-4935-8256-532404d90c0f None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:51.713751 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-30994937-534d-4935-8256-532404d90c0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:51.731707 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-cabafe88-c610-4023-854b-ee34a7afe439 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a02859127960469585afd76d1bf8f6a1', 'request_id': 'req-cabafe88-c610-4023-854b-ee34a7afe439'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:51.742343 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-cabafe88-c610-4023-854b-ee34a7afe439 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a02859127960469585afd76d1bf8f6a1', 'request_id': 'req-cabafe88-c610-4023-854b-ee34a7afe439'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:51.759480 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-cabafe88-c610-4023-854b-ee34a7afe439 admin admin] The token cache is being invalidate because project a02859127960469585afd76d1bf8f6a1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:51.759866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-cabafe88-c610-4023-854b-ee34a7afe439 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:51.760680 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-cabafe88-c610-4023-854b-ee34a7afe439 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:51.768157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 927/1852] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:51 2022] DELETE /identity/v3/projects/a02859127960469585afd76d1bf8f6a1 => generated 0 bytes in 655 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:51.769667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabafe88-c610-4023-854b-ee34a7afe439 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:51.772842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-cabafe88-c610-4023-854b-ee34a7afe439 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:51.840488 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30994937-534d-4935-8256-532404d90c0f None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:51.842501 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f090416b-2885-4109-876b-2f4a41fa85a6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:51.845169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f090416b-2885-4109-876b-2f4a41fa85a6 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:51.845169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f090416b-2885-4109-876b-2f4a41fa85a6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:51.845169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f090416b-2885-4109-876b-2f4a41fa85a6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:51.846246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f090416b-2885-4109-876b-2f4a41fa85a6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:51.868404 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f090416b-2885-4109-876b-2f4a41fa85a6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:51.870913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f090416b-2885-4109-876b-2f4a41fa85a6 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:51.890878 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 926/1853] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:51 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:25:51.894372 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabafe88-c610-4023-854b-ee34a7afe439 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:51.900541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5c94dc-151f-431a-81fc-829849037651 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:51.901184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5c94dc-151f-431a-81fc-829849037651 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:51.901458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5c94dc-151f-431a-81fc-829849037651 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:51.902353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5c94dc-151f-431a-81fc-829849037651 admin admin] PATH_INFO: `/v3/users/2401a6bcd73f44f0898ece573a0cb180` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:51.902938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5c94dc-151f-431a-81fc-829849037651 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2401a6bcd73f44f0898ece573a0cb180)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:51.905157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5c94dc-151f-431a-81fc-829849037651 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:52.141332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:52.153308 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-6a5c94dc-151f-431a-81fc-829849037651 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2401a6bcd73f44f0898ece573a0cb180'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:52.157976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f090416b-2885-4109-876b-2f4a41fa85a6 service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:52.158808 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f090416b-2885-4109-876b-2f4a41fa85a6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:52.168019 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-6a5c94dc-151f-431a-81fc-829849037651 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2401a6bcd73f44f0898ece573a0cb180'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:52.184962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f090416b-2885-4109-876b-2f4a41fa85a6 service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:52.186340 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-6a5c94dc-151f-431a-81fc-829849037651 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2401a6bcd73f44f0898ece573a0cb180'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:52.190541 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e24ba53-3350-4472-89de-d65de37e4ee0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:52.191580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e24ba53-3350-4472-89de-d65de37e4ee0 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:52.192177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e24ba53-3350-4472-89de-d65de37e4ee0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:52.197929 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e24ba53-3350-4472-89de-d65de37e4ee0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:52.197929 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7e24ba53-3350-4472-89de-d65de37e4ee0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:52.272083 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e24ba53-3350-4472-89de-d65de37e4ee0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:52.273639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e24ba53-3350-4472-89de-d65de37e4ee0 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:52.276078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-6a5c94dc-151f-431a-81fc-829849037651 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2401a6bcd73f44f0898ece573a0cb180'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:52.277048 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-6a5c94dc-151f-431a-81fc-829849037651 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2401a6bcd73f44f0898ece573a0cb180'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:52.283944 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 927/1854] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:52 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:52.298927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 928/1855] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:51 2022] DELETE /identity/v3/users/2401a6bcd73f44f0898ece573a0cb180 => generated 0 bytes in 521 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:52.896196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:52.905022 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e24ba53-3350-4472-89de-d65de37e4ee0 service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:52.917589 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e24ba53-3350-4472-89de-d65de37e4ee0 service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:53.005394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e04c7c8-4a02-4a99-8df1-0c68f0c3faf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:53.006171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e04c7c8-4a02-4a99-8df1-0c68f0c3faf7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:53.006434 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e04c7c8-4a02-4a99-8df1-0c68f0c3faf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:53.006703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e04c7c8-4a02-4a99-8df1-0c68f0c3faf7 admin admin] PATH_INFO: `/v3/projects/e0c09ae9228744f4908068a08ee77e35` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:53.007869 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e04c7c8-4a02-4a99-8df1-0c68f0c3faf7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e0c09ae9228744f4908068a08ee77e35)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:53.009838 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e04c7c8-4a02-4a99-8df1-0c68f0c3faf7 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:53.128583 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4e04c7c8-4a02-4a99-8df1-0c68f0c3faf7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e0c09ae9228744f4908068a08ee77e35', 'request_id': 'req-4e04c7c8-4a02-4a99-8df1-0c68f0c3faf7'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:53.134488 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4e04c7c8-4a02-4a99-8df1-0c68f0c3faf7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e0c09ae9228744f4908068a08ee77e35', 'request_id': 'req-4e04c7c8-4a02-4a99-8df1-0c68f0c3faf7'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:53.141802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4e04c7c8-4a02-4a99-8df1-0c68f0c3faf7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e0c09ae9228744f4908068a08ee77e35', 'request_id': 'req-4e04c7c8-4a02-4a99-8df1-0c68f0c3faf7'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:53.261629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4e04c7c8-4a02-4a99-8df1-0c68f0c3faf7 admin admin] The token cache is being invalidate because project e0c09ae9228744f4908068a08ee77e35 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:53.261629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4e04c7c8-4a02-4a99-8df1-0c68f0c3faf7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:53.261629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4e04c7c8-4a02-4a99-8df1-0c68f0c3faf7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:53.261629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 928/1856] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:52 2022] DELETE /identity/v3/projects/e0c09ae9228744f4908068a08ee77e35 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:25:53.264720 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3be2eca-7d52-4baf-9faa-3c287f768940 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:53.265314 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3be2eca-7d52-4baf-9faa-3c287f768940 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:53.265603 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3be2eca-7d52-4baf-9faa-3c287f768940 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:53.299217 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-c3be2eca-7d52-4baf-9faa-3c287f768940 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:53.301153 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c3be2eca-7d52-4baf-9faa-3c287f768940 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:53.386868 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 929/1857] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:53.396993 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a64faa5-edb7-44e1-b874-eaee5357cd4c None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:53.397540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a64faa5-edb7-44e1-b874-eaee5357cd4c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:53.397540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a64faa5-edb7-44e1-b874-eaee5357cd4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:53.427834 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-8a64faa5-edb7-44e1-b874-eaee5357cd4c None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:53.429206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8a64faa5-edb7-44e1-b874-eaee5357cd4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:53.445900 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 929/1858] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:53.450493 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3be2eca-7d52-4baf-9faa-3c287f768940 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:53.461607 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3be2eca-7d52-4baf-9faa-3c287f768940 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:53.466742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60767c03-bd02-48ae-ac69-9de25faf0c81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:53.467426 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60767c03-bd02-48ae-ac69-9de25faf0c81 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:53.467702 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60767c03-bd02-48ae-ac69-9de25faf0c81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:53.468023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60767c03-bd02-48ae-ac69-9de25faf0c81 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:53.468712 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60767c03-bd02-48ae-ac69-9de25faf0c81 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:25:53.469023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60767c03-bd02-48ae-ac69-9de25faf0c81 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:53.470864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60767c03-bd02-48ae-ac69-9de25faf0c81 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:53.484669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 930/1859] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:25:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:25:53.489753 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a64faa5-edb7-44e1-b874-eaee5357cd4c None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:53.499607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a64faa5-edb7-44e1-b874-eaee5357cd4c None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:53.505332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05104b3-bc74-4f55-8ecd-a8f2b8e4b1a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:53.505889 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05104b3-bc74-4f55-8ecd-a8f2b8e4b1a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:53.506164 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05104b3-bc74-4f55-8ecd-a8f2b8e4b1a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:53.506483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05104b3-bc74-4f55-8ecd-a8f2b8e4b1a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:53.507217 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05104b3-bc74-4f55-8ecd-a8f2b8e4b1a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:53.509060 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05104b3-bc74-4f55-8ecd-a8f2b8e4b1a2 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:53.645559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 930/1860] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:53 2022] POST /identity/v3/projects => generated 387 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:25:53.650049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60767c03-bd02-48ae-ac69-9de25faf0c81 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:53.660184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60767c03-bd02-48ae-ac69-9de25faf0c81 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:53.726956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e43029-d190-4b0f-b696-d8ef6c7f3f56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:53.727664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e43029-d190-4b0f-b696-d8ef6c7f3f56 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:53.727953 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e43029-d190-4b0f-b696-d8ef6c7f3f56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:53.728250 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e43029-d190-4b0f-b696-d8ef6c7f3f56 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:53.728896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e43029-d190-4b0f-b696-d8ef6c7f3f56 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:53.731185 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e43029-d190-4b0f-b696-d8ef6c7f3f56 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:53.871202 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 931/1861] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:53 2022] POST /identity/v3/users => generated 294 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:53.875250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05104b3-bc74-4f55-8ecd-a8f2b8e4b1a2 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:53.886613 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05104b3-bc74-4f55-8ecd-a8f2b8e4b1a2 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:53.891188 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7927df-43e1-4096-8033-00e0b65b6bbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:53.891797 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7927df-43e1-4096-8033-00e0b65b6bbb admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:53.892075 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7927df-43e1-4096-8033-00e0b65b6bbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:53.892375 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7927df-43e1-4096-8033-00e0b65b6bbb admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:53.893222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7927df-43e1-4096-8033-00e0b65b6bbb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:53.894857 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7927df-43e1-4096-8033-00e0b65b6bbb admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:53.905266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 931/1862] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:53 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:53.909235 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e43029-d190-4b0f-b696-d8ef6c7f3f56 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:53.920548 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e43029-d190-4b0f-b696-d8ef6c7f3f56 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:53.926239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0db66c0-8e5a-49ee-9417-9d1b615f3e55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:53.926898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0db66c0-8e5a-49ee-9417-9d1b615f3e55 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:53.927181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0db66c0-8e5a-49ee-9417-9d1b615f3e55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:53.927457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0db66c0-8e5a-49ee-9417-9d1b615f3e55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:53.928286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0db66c0-8e5a-49ee-9417-9d1b615f3e55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:53.929956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0db66c0-8e5a-49ee-9417-9d1b615f3e55 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:53.946339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 932/1863] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:53 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:53.947325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7927df-43e1-4096-8033-00e0b65b6bbb admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:53.959925 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7927df-43e1-4096-8033-00e0b65b6bbb admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:53.964972 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c0879e-bbb2-4c1d-9713-f2509d7b8a01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:53.965649 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c0879e-bbb2-4c1d-9713-f2509d7b8a01 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:53.965944 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c0879e-bbb2-4c1d-9713-f2509d7b8a01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:53.966274 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c0879e-bbb2-4c1d-9713-f2509d7b8a01 admin admin] PATH_INFO: `/v3/projects/753b00ca518f4954933ac93cee81a0da/users/cf55fecd148a4a70beebc2f875345d95/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:53.998934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c0879e-bbb2-4c1d-9713-f2509d7b8a01 admin admin] RBAC: Authorizing `identity:create_grant(project_id=753b00ca518f4954933ac93cee81a0da, user_id=cf55fecd148a4a70beebc2f875345d95, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:54.002625 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c0879e-bbb2-4c1d-9713-f2509d7b8a01 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:54.060637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 932/1864] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:53 2022] PUT /identity/v3/projects/753b00ca518f4954933ac93cee81a0da/users/cf55fecd148a4a70beebc2f875345d95/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:25:54.085982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb06f2cc-d78b-4115-8ce7-43f462146f14 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:54.086364 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb06f2cc-d78b-4115-8ce7-43f462146f14 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:54.086699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb06f2cc-d78b-4115-8ce7-43f462146f14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:54.115842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-fb06f2cc-d78b-4115-8ce7-43f462146f14 None None] MFA Rules not processed for user `cf55fecd148a4a70beebc2f875345d95`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:54.117128 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-fb06f2cc-d78b-4115-8ce7-43f462146f14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:54.183287 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 933/1865] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:54 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:54.187851 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c0879e-bbb2-4c1d-9713-f2509d7b8a01 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:54.253120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c0879e-bbb2-4c1d-9713-f2509d7b8a01 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:54.257866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cad372-8052-42a9-9793-2d1f33f216d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:54.258470 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cad372-8052-42a9-9793-2d1f33f216d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:54.258746 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cad372-8052-42a9-9793-2d1f33f216d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:54.259018 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cad372-8052-42a9-9793-2d1f33f216d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:54.259633 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cad372-8052-42a9-9793-2d1f33f216d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:54.261544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cad372-8052-42a9-9793-2d1f33f216d2 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:54.385513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 933/1866] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:54 2022] POST /identity/v3/projects => generated 387 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:54.386578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb06f2cc-d78b-4115-8ce7-43f462146f14 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:54.397631 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb06f2cc-d78b-4115-8ce7-43f462146f14 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:54.457417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7e7bbc-509e-4bba-aa56-7416f95252ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:54.458025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7e7bbc-509e-4bba-aa56-7416f95252ee admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:54.458302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7e7bbc-509e-4bba-aa56-7416f95252ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:54.458574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7e7bbc-509e-4bba-aa56-7416f95252ee admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:54.459208 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7e7bbc-509e-4bba-aa56-7416f95252ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:54.461155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7e7bbc-509e-4bba-aa56-7416f95252ee admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:54.584626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 934/1867] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:54 2022] POST /identity/v3/users => generated 300 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:25:54.595862 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cad372-8052-42a9-9793-2d1f33f216d2 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:54.608597 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cad372-8052-42a9-9793-2d1f33f216d2 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:54.613747 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6083b935-48c7-4225-acdf-17664d99988f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:54.614382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6083b935-48c7-4225-acdf-17664d99988f admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:54.614687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6083b935-48c7-4225-acdf-17664d99988f admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:54.614965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6083b935-48c7-4225-acdf-17664d99988f admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:54.615614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6083b935-48c7-4225-acdf-17664d99988f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:54.617260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6083b935-48c7-4225-acdf-17664d99988f admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:54.629998 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 934/1868] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:54 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:54.634007 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7e7bbc-509e-4bba-aa56-7416f95252ee admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:54.645097 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7e7bbc-509e-4bba-aa56-7416f95252ee admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:54.649504 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:54.652000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67f232f-f01c-49bf-a9b1-b22f31127cd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:54.657911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67f232f-f01c-49bf-a9b1-b22f31127cd4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:54.657911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67f232f-f01c-49bf-a9b1-b22f31127cd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:54.657911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67f232f-f01c-49bf-a9b1-b22f31127cd4 admin admin] PATH_INFO: `/v3/projects/ad1870fb0ee04cee8a5796db748d4741/users/a2007c6863c843c598acc274aaf1e7b4/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:54.669885 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67f232f-f01c-49bf-a9b1-b22f31127cd4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad1870fb0ee04cee8a5796db748d4741, user_id=a2007c6863c843c598acc274aaf1e7b4, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:54.672582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67f232f-f01c-49bf-a9b1-b22f31127cd4 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:54.705713 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:54.764316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 935/1869] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:54 2022] PUT /identity/v3/projects/ad1870fb0ee04cee8a5796db748d4741/users/a2007c6863c843c598acc274aaf1e7b4/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:54.768608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6083b935-48c7-4225-acdf-17664d99988f admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:54.780300 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6083b935-48c7-4225-acdf-17664d99988f admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:54.843283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4b97bd-3cdb-49df-ace0-bdb7a95bdd8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:54.843867 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4b97bd-3cdb-49df-ace0-bdb7a95bdd8d admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:54.844112 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4b97bd-3cdb-49df-ace0-bdb7a95bdd8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:54.844520 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4b97bd-3cdb-49df-ace0-bdb7a95bdd8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:54.845156 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4b97bd-3cdb-49df-ace0-bdb7a95bdd8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:54.846746 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4b97bd-3cdb-49df-ace0-bdb7a95bdd8d admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:54.857932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 935/1870] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:54 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:25:54.862165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67f232f-f01c-49bf-a9b1-b22f31127cd4 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:54.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67f232f-f01c-49bf-a9b1-b22f31127cd4 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:54.878108 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9416a412-cf9f-4a9e-affa-ec28b1c89975 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:54.878813 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9416a412-cf9f-4a9e-affa-ec28b1c89975 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:54.879088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9416a412-cf9f-4a9e-affa-ec28b1c89975 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:54.879358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9416a412-cf9f-4a9e-affa-ec28b1c89975 admin admin] PATH_INFO: `/v3/projects/ad1870fb0ee04cee8a5796db748d4741/users/a2007c6863c843c598acc274aaf1e7b4/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:54.881420 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9416a412-cf9f-4a9e-affa-ec28b1c89975 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad1870fb0ee04cee8a5796db748d4741, user_id=a2007c6863c843c598acc274aaf1e7b4, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:54.884062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9416a412-cf9f-4a9e-affa-ec28b1c89975 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:54.895414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 936/1871] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:54 2022] PUT /identity/v3/projects/ad1870fb0ee04cee8a5796db748d4741/users/a2007c6863c843c598acc274aaf1e7b4/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:54.906537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff295f0-45c2-4829-8df5-0d367c9ac88c None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:54.906816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff295f0-45c2-4829-8df5-0d367c9ac88c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:54.907087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff295f0-45c2-4829-8df5-0d367c9ac88c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:54.935882 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-9ff295f0-45c2-4829-8df5-0d367c9ac88c None None] MFA Rules not processed for user `a2007c6863c843c598acc274aaf1e7b4`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:54.937142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9ff295f0-45c2-4829-8df5-0d367c9ac88c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:55.005737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 936/1872] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:54 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:55.024425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9416a412-cf9f-4a9e-affa-ec28b1c89975 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:55.026166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9416a412-cf9f-4a9e-affa-ec28b1c89975 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:55.087503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9416a412-cf9f-4a9e-affa-ec28b1c89975 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:55.091502 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17c9bde-ab6c-4beb-8058-2ae7a82d567f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:55.098277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17c9bde-ab6c-4beb-8058-2ae7a82d567f service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:55.098277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17c9bde-ab6c-4beb-8058-2ae7a82d567f service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:55.098277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17c9bde-ab6c-4beb-8058-2ae7a82d567f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:55.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17c9bde-ab6c-4beb-8058-2ae7a82d567f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:55.114189 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17c9bde-ab6c-4beb-8058-2ae7a82d567f service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:55.125954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 937/1873] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:55 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:55.213720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff295f0-45c2-4829-8df5-0d367c9ac88c None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:55.280885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff295f0-45c2-4829-8df5-0d367c9ac88c None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:55.285873 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409395a4-fcb1-4fe6-ba8c-778fef227d81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:55.289690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409395a4-fcb1-4fe6-ba8c-778fef227d81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:55.289690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409395a4-fcb1-4fe6-ba8c-778fef227d81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:55.289690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409395a4-fcb1-4fe6-ba8c-778fef227d81 admin admin] PATH_INFO: `/v3/users/cf55fecd148a4a70beebc2f875345d95` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:55.289690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409395a4-fcb1-4fe6-ba8c-778fef227d81 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cf55fecd148a4a70beebc2f875345d95)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:55.290406 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409395a4-fcb1-4fe6-ba8c-778fef227d81 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:55.389326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:55.536411 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-409395a4-fcb1-4fe6-ba8c-778fef227d81 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cf55fecd148a4a70beebc2f875345d95'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:55.543951 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-409395a4-fcb1-4fe6-ba8c-778fef227d81 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cf55fecd148a4a70beebc2f875345d95'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:55.544666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-409395a4-fcb1-4fe6-ba8c-778fef227d81 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cf55fecd148a4a70beebc2f875345d95'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:55.558006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-409395a4-fcb1-4fe6-ba8c-778fef227d81 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cf55fecd148a4a70beebc2f875345d95'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:55.665251 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-409395a4-fcb1-4fe6-ba8c-778fef227d81 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cf55fecd148a4a70beebc2f875345d95'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:55.674886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 937/1874] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:55 2022] DELETE /identity/v3/users/cf55fecd148a4a70beebc2f875345d95 => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:55.677282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88d8ebc-b9e2-41d1-9413-e3a8aa4bfd9a None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:55.677565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88d8ebc-b9e2-41d1-9413-e3a8aa4bfd9a None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:55.677835 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88d8ebc-b9e2-41d1-9413-e3a8aa4bfd9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:55.696619 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-d88d8ebc-b9e2-41d1-9413-e3a8aa4bfd9a None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:55.697872 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-d88d8ebc-b9e2-41d1-9413-e3a8aa4bfd9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:55.762246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 938/1875] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:55.774928 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409395a4-fcb1-4fe6-ba8c-778fef227d81 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:55.776374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-409395a4-fcb1-4fe6-ba8c-778fef227d81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:55.848487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409395a4-fcb1-4fe6-ba8c-778fef227d81 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:55.852159 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf6d157-fd78-4729-a368-5feba8edeedf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:55.852810 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf6d157-fd78-4729-a368-5feba8edeedf service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:55.853939 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf6d157-fd78-4729-a368-5feba8edeedf service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:55.853939 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf6d157-fd78-4729-a368-5feba8edeedf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:55.863529 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf6d157-fd78-4729-a368-5feba8edeedf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:55.865128 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf6d157-fd78-4729-a368-5feba8edeedf service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:55.874787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 938/1876] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:25:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:25:55.906256 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:56.146864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:56.474649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88d8ebc-b9e2-41d1-9413-e3a8aa4bfd9a None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:56.476262 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-d88d8ebc-b9e2-41d1-9413-e3a8aa4bfd9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:56.489482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88d8ebc-b9e2-41d1-9413-e3a8aa4bfd9a None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:56.495451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea34aa33-21d7-4059-81f5-482c7835bdb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:56.496116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea34aa33-21d7-4059-81f5-482c7835bdb0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:56.496405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea34aa33-21d7-4059-81f5-482c7835bdb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:56.496674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea34aa33-21d7-4059-81f5-482c7835bdb0 admin admin] PATH_INFO: `/v3/projects/753b00ca518f4954933ac93cee81a0da` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:56.498579 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea34aa33-21d7-4059-81f5-482c7835bdb0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=753b00ca518f4954933ac93cee81a0da)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:56.499798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea34aa33-21d7-4059-81f5-482c7835bdb0 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:56.638557 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ea34aa33-21d7-4059-81f5-482c7835bdb0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '753b00ca518f4954933ac93cee81a0da', 'request_id': 'req-ea34aa33-21d7-4059-81f5-482c7835bdb0'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:56.791666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:56.881637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ea34aa33-21d7-4059-81f5-482c7835bdb0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '753b00ca518f4954933ac93cee81a0da', 'request_id': 'req-ea34aa33-21d7-4059-81f5-482c7835bdb0'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:56.889370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ea34aa33-21d7-4059-81f5-482c7835bdb0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '753b00ca518f4954933ac93cee81a0da', 'request_id': 'req-ea34aa33-21d7-4059-81f5-482c7835bdb0'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:56.914445 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ea34aa33-21d7-4059-81f5-482c7835bdb0 admin admin] The token cache is being invalidate because project 753b00ca518f4954933ac93cee81a0da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:56.914445 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ea34aa33-21d7-4059-81f5-482c7835bdb0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:56.914445 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ea34aa33-21d7-4059-81f5-482c7835bdb0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:56.914445 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 939/1877] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:56 2022] DELETE /identity/v3/projects/753b00ca518f4954933ac93cee81a0da => generated 0 bytes in 435 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:56.914926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf6d157-fd78-4729-a368-5feba8edeedf service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:56.923018 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf6d157-fd78-4729-a368-5feba8edeedf service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:56.983442 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5cb982-8c7d-4744-b649-41b6b5747a58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:56.984518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5cb982-8c7d-4744-b649-41b6b5747a58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:56.984872 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5cb982-8c7d-4744-b649-41b6b5747a58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:56.985230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5cb982-8c7d-4744-b649-41b6b5747a58 admin admin] PATH_INFO: `/v3/users/a2007c6863c843c598acc274aaf1e7b4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:56.986506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5cb982-8c7d-4744-b649-41b6b5747a58 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a2007c6863c843c598acc274aaf1e7b4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:56.989046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5cb982-8c7d-4744-b649-41b6b5747a58 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:57.219263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1f5cb982-8c7d-4744-b649-41b6b5747a58 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a2007c6863c843c598acc274aaf1e7b4'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:57.227641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1f5cb982-8c7d-4744-b649-41b6b5747a58 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a2007c6863c843c598acc274aaf1e7b4'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:57.228552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1f5cb982-8c7d-4744-b649-41b6b5747a58 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a2007c6863c843c598acc274aaf1e7b4'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:57.253163 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1f5cb982-8c7d-4744-b649-41b6b5747a58 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a2007c6863c843c598acc274aaf1e7b4'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:57.351835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1f5cb982-8c7d-4744-b649-41b6b5747a58 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a2007c6863c843c598acc274aaf1e7b4'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:57.363984 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 939/1878] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:25:56 2022] DELETE /identity/v3/users/a2007c6863c843c598acc274aaf1e7b4 => generated 0 bytes in 448 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:57.548725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:57.662261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:58.039846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea34aa33-21d7-4059-81f5-482c7835bdb0 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:58.041772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ea34aa33-21d7-4059-81f5-482c7835bdb0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:58.061895 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea34aa33-21d7-4059-81f5-482c7835bdb0 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:58.139649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32edbd8a-6192-47fa-aa2b-9cde8192152e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:58.140320 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32edbd8a-6192-47fa-aa2b-9cde8192152e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:58.140600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32edbd8a-6192-47fa-aa2b-9cde8192152e admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:58.140875 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32edbd8a-6192-47fa-aa2b-9cde8192152e admin admin] PATH_INFO: `/v3/projects/ad1870fb0ee04cee8a5796db748d4741` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:58.142101 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32edbd8a-6192-47fa-aa2b-9cde8192152e admin admin] RBAC: Authorizing `identity:delete_project(project_id=ad1870fb0ee04cee8a5796db748d4741)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:58.144560 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32edbd8a-6192-47fa-aa2b-9cde8192152e admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:58.284779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-32edbd8a-6192-47fa-aa2b-9cde8192152e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ad1870fb0ee04cee8a5796db748d4741', 'request_id': 'req-32edbd8a-6192-47fa-aa2b-9cde8192152e'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:58.351936 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-32edbd8a-6192-47fa-aa2b-9cde8192152e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ad1870fb0ee04cee8a5796db748d4741', 'request_id': 'req-32edbd8a-6192-47fa-aa2b-9cde8192152e'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:58.358384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-32edbd8a-6192-47fa-aa2b-9cde8192152e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ad1870fb0ee04cee8a5796db748d4741', 'request_id': 'req-32edbd8a-6192-47fa-aa2b-9cde8192152e'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:58.372828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-32edbd8a-6192-47fa-aa2b-9cde8192152e admin admin] The token cache is being invalidate because project ad1870fb0ee04cee8a5796db748d4741 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:25:58.373213 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-32edbd8a-6192-47fa-aa2b-9cde8192152e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:58.374022 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-32edbd8a-6192-47fa-aa2b-9cde8192152e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:25:58.376090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 940/1879] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:25:58 2022] DELETE /identity/v3/projects/ad1870fb0ee04cee8a5796db748d4741 => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:25:58.382692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a856898-6850-4038-b99f-e1d5b5cfa620 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:58.383201 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a856898-6850-4038-b99f-e1d5b5cfa620 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:58.383287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a856898-6850-4038-b99f-e1d5b5cfa620 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:58.403573 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-5a856898-6850-4038-b99f-e1d5b5cfa620 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:58.404822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5a856898-6850-4038-b99f-e1d5b5cfa620 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:58.450609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:58.488331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 940/1880] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:58.496150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bcbd6a-643b-47a9-8ab8-b4234b5eedd2 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:58.496447 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bcbd6a-643b-47a9-8ab8-b4234b5eedd2 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:58.496858 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bcbd6a-643b-47a9-8ab8-b4234b5eedd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:58.521485 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-88bcbd6a-643b-47a9-8ab8-b4234b5eedd2 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:58.523310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-88bcbd6a-643b-47a9-8ab8-b4234b5eedd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:58.541736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 941/1881] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:58.542122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a856898-6850-4038-b99f-e1d5b5cfa620 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:58.547895 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a856898-6850-4038-b99f-e1d5b5cfa620 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:58.555089 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213b1ad2-770c-4778-9c28-e2b0485431fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:58.555684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213b1ad2-770c-4778-9c28-e2b0485431fd admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:58.555959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213b1ad2-770c-4778-9c28-e2b0485431fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:58.556429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213b1ad2-770c-4778-9c28-e2b0485431fd admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:58.558142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213b1ad2-770c-4778-9c28-e2b0485431fd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:25:58.558142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213b1ad2-770c-4778-9c28-e2b0485431fd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:58.559878 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213b1ad2-770c-4778-9c28-e2b0485431fd admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:58.574417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 941/1882] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:25:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:25:58.578915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bcbd6a-643b-47a9-8ab8-b4234b5eedd2 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:58.588564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bcbd6a-643b-47a9-8ab8-b4234b5eedd2 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:58.593933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53be0e17-4c6e-4ee8-82c1-610c7431e425 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:58.594543 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be0e17-4c6e-4ee8-82c1-610c7431e425 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:58.594817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be0e17-4c6e-4ee8-82c1-610c7431e425 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:58.595085 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be0e17-4c6e-4ee8-82c1-610c7431e425 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:58.595924 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53be0e17-4c6e-4ee8-82c1-610c7431e425 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:58.597949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53be0e17-4c6e-4ee8-82c1-610c7431e425 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:58.720480 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 942/1883] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:58 2022] POST /identity/v3/projects => generated 365 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:58.725318 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213b1ad2-770c-4778-9c28-e2b0485431fd admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:58.736221 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213b1ad2-770c-4778-9c28-e2b0485431fd admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:58.802336 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737d3f2b-7c6a-4d57-bbeb-1c2164f630c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:58.803728 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737d3f2b-7c6a-4d57-bbeb-1c2164f630c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:58.804253 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737d3f2b-7c6a-4d57-bbeb-1c2164f630c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:58.804731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737d3f2b-7c6a-4d57-bbeb-1c2164f630c7 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:58.805903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737d3f2b-7c6a-4d57-bbeb-1c2164f630c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:58.810226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737d3f2b-7c6a-4d57-bbeb-1c2164f630c7 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:58.888823 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:58.907329 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 942/1884] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:58 2022] POST /identity/v3/users => generated 283 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:58.908720 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53be0e17-4c6e-4ee8-82c1-610c7431e425 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:58.922003 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53be0e17-4c6e-4ee8-82c1-610c7431e425 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:58.930826 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02efd0f-b6de-4790-8fbc-00f65fe0f484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:58.931474 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02efd0f-b6de-4790-8fbc-00f65fe0f484 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:58.931759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02efd0f-b6de-4790-8fbc-00f65fe0f484 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:58.932298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02efd0f-b6de-4790-8fbc-00f65fe0f484 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:58.933241 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02efd0f-b6de-4790-8fbc-00f65fe0f484 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:58.935848 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02efd0f-b6de-4790-8fbc-00f65fe0f484 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:58.951859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 943/1885] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:58 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:25:58.957381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737d3f2b-7c6a-4d57-bbeb-1c2164f630c7 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:59.001766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737d3f2b-7c6a-4d57-bbeb-1c2164f630c7 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:59.016525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5eb5f65-6f89-4df5-8499-7368f35f1d4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:59.017157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5eb5f65-6f89-4df5-8499-7368f35f1d4a admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:59.017439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5eb5f65-6f89-4df5-8499-7368f35f1d4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:59.018150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5eb5f65-6f89-4df5-8499-7368f35f1d4a admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:59.019275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5eb5f65-6f89-4df5-8499-7368f35f1d4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:59.021160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5eb5f65-6f89-4df5-8499-7368f35f1d4a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:59.034931 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 943/1886] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:58 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:25:59.037837 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02efd0f-b6de-4790-8fbc-00f65fe0f484 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:59.063586 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02efd0f-b6de-4790-8fbc-00f65fe0f484 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:59.063586 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194218fd-fc34-485d-b2d7-a891a0f12ba6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:59.064232 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194218fd-fc34-485d-b2d7-a891a0f12ba6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:59.064513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194218fd-fc34-485d-b2d7-a891a0f12ba6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:59.064784 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194218fd-fc34-485d-b2d7-a891a0f12ba6 admin admin] PATH_INFO: `/v3/projects/02ec9f9599c9451fabc232e5cb0933b1/users/f855040e6fd74026854fb997d888df17/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:59.079948 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194218fd-fc34-485d-b2d7-a891a0f12ba6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=02ec9f9599c9451fabc232e5cb0933b1, user_id=f855040e6fd74026854fb997d888df17, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:59.083183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194218fd-fc34-485d-b2d7-a891a0f12ba6 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:59.209132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 944/1887] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:59 2022] PUT /identity/v3/projects/02ec9f9599c9451fabc232e5cb0933b1/users/f855040e6fd74026854fb997d888df17/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:25:59.219694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d08c3e-5ca6-43f9-a594-fb6bb5366e43 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:59.220019 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d08c3e-5ca6-43f9-a594-fb6bb5366e43 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:59.220399 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d08c3e-5ca6-43f9-a594-fb6bb5366e43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:59.254241 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-71d08c3e-5ca6-43f9-a594-fb6bb5366e43 None None] MFA Rules not processed for user `f855040e6fd74026854fb997d888df17`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:25:59.255449 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-71d08c3e-5ca6-43f9-a594-fb6bb5366e43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:25:59.324785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 944/1888] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:25:59 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:25:59.329763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194218fd-fc34-485d-b2d7-a891a0f12ba6 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:59.410184 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194218fd-fc34-485d-b2d7-a891a0f12ba6 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:59.415283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07eed722-994e-4fd5-865f-e8827528f5d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:59.415897 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eed722-994e-4fd5-865f-e8827528f5d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:59.416201 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eed722-994e-4fd5-865f-e8827528f5d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:59.416471 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eed722-994e-4fd5-865f-e8827528f5d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:59.417093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07eed722-994e-4fd5-865f-e8827528f5d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:59.419018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07eed722-994e-4fd5-865f-e8827528f5d9 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:59.583587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 945/1889] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:25:59 2022] POST /identity/v3/projects => generated 365 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:59.585096 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d08c3e-5ca6-43f9-a594-fb6bb5366e43 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:59.605415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d08c3e-5ca6-43f9-a594-fb6bb5366e43 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:59.730743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e24ff2-6245-4a4c-a298-fb587bcdbc26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:59.731370 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e24ff2-6245-4a4c-a298-fb587bcdbc26 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:59.731649 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e24ff2-6245-4a4c-a298-fb587bcdbc26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:59.734084 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e24ff2-6245-4a4c-a298-fb587bcdbc26 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:59.734084 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e24ff2-6245-4a4c-a298-fb587bcdbc26 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:59.734539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e24ff2-6245-4a4c-a298-fb587bcdbc26 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:59.814273 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:DELETE=17,keystone:SELECT=81,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:25:59.894868 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 945/1890] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:25:59 2022] POST /identity/v3/users => generated 289 bytes in 311 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:25:59.898744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07eed722-994e-4fd5-865f-e8827528f5d9 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:59.909993 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07eed722-994e-4fd5-865f-e8827528f5d9 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:59.915616 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2721036-37f0-4115-b515-6204b57d27fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:59.916424 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2721036-37f0-4115-b515-6204b57d27fb admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:59.916793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2721036-37f0-4115-b515-6204b57d27fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:59.917154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2721036-37f0-4115-b515-6204b57d27fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:59.918043 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2721036-37f0-4115-b515-6204b57d27fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:59.920274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2721036-37f0-4115-b515-6204b57d27fb admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:25:59.933299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 946/1891] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:25:59 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:25:59.937522 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e24ff2-6245-4a4c-a298-fb587bcdbc26 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:25:59.949680 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e24ff2-6245-4a4c-a298-fb587bcdbc26 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:25:59.954522 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f4cb04-4489-4560-96d4-bac7290d9f7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:25:59.955224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f4cb04-4489-4560-96d4-bac7290d9f7b admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:25:59.955487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f4cb04-4489-4560-96d4-bac7290d9f7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:25:59.955771 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f4cb04-4489-4560-96d4-bac7290d9f7b admin admin] PATH_INFO: `/v3/projects/936fb875ffe442cd82df135b92abb639/users/ba7b6a5ee9b84d4ea25b1cfc860ca949/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:25:59.973294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f4cb04-4489-4560-96d4-bac7290d9f7b admin admin] RBAC: Authorizing `identity:create_grant(project_id=936fb875ffe442cd82df135b92abb639, user_id=ba7b6a5ee9b84d4ea25b1cfc860ca949, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:25:59.975991 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f4cb04-4489-4560-96d4-bac7290d9f7b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:00.105866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 946/1892] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:25:59 2022] PUT /identity/v3/projects/936fb875ffe442cd82df135b92abb639/users/ba7b6a5ee9b84d4ea25b1cfc860ca949/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:00.110035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2721036-37f0-4115-b515-6204b57d27fb admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:00.121411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2721036-37f0-4115-b515-6204b57d27fb admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:00.191789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc30015-0b14-4608-bbf7-2cc49ec1b244 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:00.192427 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc30015-0b14-4608-bbf7-2cc49ec1b244 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:00.192700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc30015-0b14-4608-bbf7-2cc49ec1b244 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:00.193024 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc30015-0b14-4608-bbf7-2cc49ec1b244 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:00.194059 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc30015-0b14-4608-bbf7-2cc49ec1b244 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:00.195779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc30015-0b14-4608-bbf7-2cc49ec1b244 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:00.206744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 947/1893] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:00 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:00.211854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f4cb04-4489-4560-96d4-bac7290d9f7b admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:00.230705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f4cb04-4489-4560-96d4-bac7290d9f7b admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:00.236003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05639cc8-276c-4ff2-bf08-2fd872c505be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:00.236716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05639cc8-276c-4ff2-bf08-2fd872c505be admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:00.237047 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05639cc8-276c-4ff2-bf08-2fd872c505be admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:00.237508 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05639cc8-276c-4ff2-bf08-2fd872c505be admin admin] PATH_INFO: `/v3/projects/936fb875ffe442cd82df135b92abb639/users/ba7b6a5ee9b84d4ea25b1cfc860ca949/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:00.240086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05639cc8-276c-4ff2-bf08-2fd872c505be admin admin] RBAC: Authorizing `identity:create_grant(project_id=936fb875ffe442cd82df135b92abb639, user_id=ba7b6a5ee9b84d4ea25b1cfc860ca949, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:00.242779 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05639cc8-276c-4ff2-bf08-2fd872c505be admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:00.256143 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 947/1894] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:00 2022] PUT /identity/v3/projects/936fb875ffe442cd82df135b92abb639/users/ba7b6a5ee9b84d4ea25b1cfc860ca949/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:00.266996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3789ff1f-0dd3-4b66-89ca-d1174fd6f9b2 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:00.267282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3789ff1f-0dd3-4b66-89ca-d1174fd6f9b2 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:00.267559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3789ff1f-0dd3-4b66-89ca-d1174fd6f9b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:00.305325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-3789ff1f-0dd3-4b66-89ca-d1174fd6f9b2 None None] MFA Rules not processed for user `ba7b6a5ee9b84d4ea25b1cfc860ca949`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:00.307709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3789ff1f-0dd3-4b66-89ca-d1174fd6f9b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:00.391015 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 948/1895] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:00 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:00.402904 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05639cc8-276c-4ff2-bf08-2fd872c505be admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:00.404653 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-05639cc8-276c-4ff2-bf08-2fd872c505be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:00.480772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05639cc8-276c-4ff2-bf08-2fd872c505be admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:00.484718 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b109714d-bc3d-4777-a5e2-f5df4136f775 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:00.485365 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b109714d-bc3d-4777-a5e2-f5df4136f775 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:00.485646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b109714d-bc3d-4777-a5e2-f5df4136f775 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:00.486115 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b109714d-bc3d-4777-a5e2-f5df4136f775 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:00.498094 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b109714d-bc3d-4777-a5e2-f5df4136f775 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:00.499533 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b109714d-bc3d-4777-a5e2-f5df4136f775 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:00.510260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 948/1896] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:00 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:00.864266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:00.893210 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3789ff1f-0dd3-4b66-89ca-d1174fd6f9b2 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:00.893210 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3789ff1f-0dd3-4b66-89ca-d1174fd6f9b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:00.902930 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3789ff1f-0dd3-4b66-89ca-d1174fd6f9b2 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:00.907298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78226eff-0109-425a-9984-dd0f247fefbf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:00.908126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78226eff-0109-425a-9984-dd0f247fefbf service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:00.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78226eff-0109-425a-9984-dd0f247fefbf service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:00.908950 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78226eff-0109-425a-9984-dd0f247fefbf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:00.987149 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78226eff-0109-425a-9984-dd0f247fefbf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:00.988488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78226eff-0109-425a-9984-dd0f247fefbf service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:00.999051 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 949/1897] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:00 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:01.556227 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b109714d-bc3d-4777-a5e2-f5df4136f775 service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:01.557779 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b109714d-bc3d-4777-a5e2-f5df4136f775 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:01.570759 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=84,keystone:DELETE=10,keystone:INSERT=6 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:01.630327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b109714d-bc3d-4777-a5e2-f5df4136f775 service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:01.634949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cc2416-1d36-495b-8227-0bc49d23cbcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:01.635748 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cc2416-1d36-495b-8227-0bc49d23cbcc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:01.639176 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cc2416-1d36-495b-8227-0bc49d23cbcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:01.639176 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cc2416-1d36-495b-8227-0bc49d23cbcc admin admin] PATH_INFO: `/v3/users/1064255db4414ac0b28481508ebb4d45` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:01.639176 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cc2416-1d36-495b-8227-0bc49d23cbcc admin admin] RBAC: Authorizing `identity:delete_user(user_id=1064255db4414ac0b28481508ebb4d45)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:01.639601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cc2416-1d36-495b-8227-0bc49d23cbcc admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:01.940895 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-25cc2416-1d36-495b-8227-0bc49d23cbcc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1064255db4414ac0b28481508ebb4d45'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:01.952334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-25cc2416-1d36-495b-8227-0bc49d23cbcc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1064255db4414ac0b28481508ebb4d45'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:01.953276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-25cc2416-1d36-495b-8227-0bc49d23cbcc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1064255db4414ac0b28481508ebb4d45'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:01.973024 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-25cc2416-1d36-495b-8227-0bc49d23cbcc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1064255db4414ac0b28481508ebb4d45'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:01.979867 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78226eff-0109-425a-9984-dd0f247fefbf service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:01.985306 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-78226eff-0109-425a-9984-dd0f247fefbf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:02.003540 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78226eff-0109-425a-9984-dd0f247fefbf service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:02.009736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36253539-11d6-4eab-a837-15c912039608 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:02.010492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36253539-11d6-4eab-a837-15c912039608 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:02.010765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36253539-11d6-4eab-a837-15c912039608 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:02.011215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36253539-11d6-4eab-a837-15c912039608 admin admin] PATH_INFO: `/v3/users/f80525e5e3c0428291a95c5e31f8d3bc` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:02.012434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36253539-11d6-4eab-a837-15c912039608 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f80525e5e3c0428291a95c5e31f8d3bc)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:02.014371 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36253539-11d6-4eab-a837-15c912039608 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:02.114372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-25cc2416-1d36-495b-8227-0bc49d23cbcc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1064255db4414ac0b28481508ebb4d45'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:02.122105 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 949/1898] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:01 2022] DELETE /identity/v3/users/1064255db4414ac0b28481508ebb4d45 => generated 0 bytes in 566 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:02.126391 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1bf231-1353-472a-9f23-be042c7244c7 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:02.127549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1bf231-1353-472a-9f23-be042c7244c7 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:02.127549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1bf231-1353-472a-9f23-be042c7244c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:02.146483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-7c1bf231-1353-472a-9f23-be042c7244c7 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:02.147778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7c1bf231-1353-472a-9f23-be042c7244c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:02.209346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 950/1899] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:02.219285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1bf231-1353-472a-9f23-be042c7244c7 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:02.220771 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7c1bf231-1353-472a-9f23-be042c7244c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:02.281552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1bf231-1353-472a-9f23-be042c7244c7 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:02.286511 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df241ab3-2d0a-40c9-8e1e-3bb9b038f569 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:02.287122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df241ab3-2d0a-40c9-8e1e-3bb9b038f569 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:02.287663 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df241ab3-2d0a-40c9-8e1e-3bb9b038f569 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:02.287933 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df241ab3-2d0a-40c9-8e1e-3bb9b038f569 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:02.297388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df241ab3-2d0a-40c9-8e1e-3bb9b038f569 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:02.298841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df241ab3-2d0a-40c9-8e1e-3bb9b038f569 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:02.308855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 951/1900] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:26:02.424647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-36253539-11d6-4eab-a837-15c912039608 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f80525e5e3c0428291a95c5e31f8d3bc'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:02.432035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-36253539-11d6-4eab-a837-15c912039608 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f80525e5e3c0428291a95c5e31f8d3bc'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:02.444817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-36253539-11d6-4eab-a837-15c912039608 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f80525e5e3c0428291a95c5e31f8d3bc'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:02.582083 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-36253539-11d6-4eab-a837-15c912039608 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f80525e5e3c0428291a95c5e31f8d3bc'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:02.582838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-36253539-11d6-4eab-a837-15c912039608 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f80525e5e3c0428291a95c5e31f8d3bc'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:02.597188 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 950/1901] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:01 2022] DELETE /identity/v3/users/f80525e5e3c0428291a95c5e31f8d3bc => generated 0 bytes in 611 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:02.597394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc2aa1e-1864-44ad-94fb-c9ceb4676664 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:02.597394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc2aa1e-1864-44ad-94fb-c9ceb4676664 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:02.597394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc2aa1e-1864-44ad-94fb-c9ceb4676664 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:02.629966 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-bbc2aa1e-1864-44ad-94fb-c9ceb4676664 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:02.633434 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-bbc2aa1e-1864-44ad-94fb-c9ceb4676664 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:02.740020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 952/1902] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:02.747540 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36253539-11d6-4eab-a837-15c912039608 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:02.749141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-36253539-11d6-4eab-a837-15c912039608 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:02.816920 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36253539-11d6-4eab-a837-15c912039608 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:02.821632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45feaee-184e-476a-8e6f-aa3e3e91acd1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:02.822518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45feaee-184e-476a-8e6f-aa3e3e91acd1 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:02.823073 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45feaee-184e-476a-8e6f-aa3e3e91acd1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:02.823495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45feaee-184e-476a-8e6f-aa3e3e91acd1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:02.833198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:02.852179 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45feaee-184e-476a-8e6f-aa3e3e91acd1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:02.855288 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45feaee-184e-476a-8e6f-aa3e3e91acd1 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:02.870815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 951/1903] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:03.156563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc2aa1e-1864-44ad-94fb-c9ceb4676664 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:03.158700 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-bbc2aa1e-1864-44ad-94fb-c9ceb4676664 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:03.178213 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc2aa1e-1864-44ad-94fb-c9ceb4676664 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:03.184081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918acbf9-1a8e-4bca-83be-e38a8a002b73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:03.184962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918acbf9-1a8e-4bca-83be-e38a8a002b73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:03.185370 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918acbf9-1a8e-4bca-83be-e38a8a002b73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:03.185736 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918acbf9-1a8e-4bca-83be-e38a8a002b73 admin admin] PATH_INFO: `/v3/projects/0256ec2f06b249cfb4ab2308adaa8713` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:03.189146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918acbf9-1a8e-4bca-83be-e38a8a002b73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0256ec2f06b249cfb4ab2308adaa8713)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:03.204413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918acbf9-1a8e-4bca-83be-e38a8a002b73 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:03.405805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-918acbf9-1a8e-4bca-83be-e38a8a002b73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0256ec2f06b249cfb4ab2308adaa8713', 'request_id': 'req-918acbf9-1a8e-4bca-83be-e38a8a002b73'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:03.413900 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-918acbf9-1a8e-4bca-83be-e38a8a002b73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0256ec2f06b249cfb4ab2308adaa8713', 'request_id': 'req-918acbf9-1a8e-4bca-83be-e38a8a002b73'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:03.423960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-918acbf9-1a8e-4bca-83be-e38a8a002b73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0256ec2f06b249cfb4ab2308adaa8713', 'request_id': 'req-918acbf9-1a8e-4bca-83be-e38a8a002b73'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:03.451673 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45feaee-184e-476a-8e6f-aa3e3e91acd1 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:03.458644 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e45feaee-184e-476a-8e6f-aa3e3e91acd1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:03.503760 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45feaee-184e-476a-8e6f-aa3e3e91acd1 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:03.558716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-918acbf9-1a8e-4bca-83be-e38a8a002b73 admin admin] The token cache is being invalidate because project 0256ec2f06b249cfb4ab2308adaa8713 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:03.559001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-918acbf9-1a8e-4bca-83be-e38a8a002b73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:03.562567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-918acbf9-1a8e-4bca-83be-e38a8a002b73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:03.566480 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 953/1904] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:03 2022] DELETE /identity/v3/projects/0256ec2f06b249cfb4ab2308adaa8713 => generated 0 bytes in 411 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:03.574344 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918acbf9-1a8e-4bca-83be-e38a8a002b73 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:03.576438 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-918acbf9-1a8e-4bca-83be-e38a8a002b73 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:03.619947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9febb3-12b5-4fe0-908b-bfb88842900f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:03.620804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9febb3-12b5-4fe0-908b-bfb88842900f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:03.621160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9febb3-12b5-4fe0-908b-bfb88842900f admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:03.621519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9febb3-12b5-4fe0-908b-bfb88842900f admin admin] PATH_INFO: `/v3/projects/0d062704975147a3b6b674b4e71f8e61` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:03.622875 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9febb3-12b5-4fe0-908b-bfb88842900f admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d062704975147a3b6b674b4e71f8e61)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:03.625150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9febb3-12b5-4fe0-908b-bfb88842900f admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:03.665430 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918acbf9-1a8e-4bca-83be-e38a8a002b73 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:03.670587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407f310b-74f5-406c-b03a-a4c2818f38f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:03.671239 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407f310b-74f5-406c-b03a-a4c2818f38f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:03.671518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407f310b-74f5-406c-b03a-a4c2818f38f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:03.671793 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407f310b-74f5-406c-b03a-a4c2818f38f6 admin admin] PATH_INFO: `/v3/users/427969920afc46ee8adef66c927073be` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:03.672963 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407f310b-74f5-406c-b03a-a4c2818f38f6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=427969920afc46ee8adef66c927073be)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:03.675506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407f310b-74f5-406c-b03a-a4c2818f38f6 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:03.853227 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0e9febb3-12b5-4fe0-908b-bfb88842900f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d062704975147a3b6b674b4e71f8e61', 'request_id': 'req-0e9febb3-12b5-4fe0-908b-bfb88842900f'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:04.030993 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0e9febb3-12b5-4fe0-908b-bfb88842900f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d062704975147a3b6b674b4e71f8e61', 'request_id': 'req-0e9febb3-12b5-4fe0-908b-bfb88842900f'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:04.043205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0e9febb3-12b5-4fe0-908b-bfb88842900f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d062704975147a3b6b674b4e71f8e61', 'request_id': 'req-0e9febb3-12b5-4fe0-908b-bfb88842900f'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:04.058425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-407f310b-74f5-406c-b03a-a4c2818f38f6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '427969920afc46ee8adef66c927073be'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:04.060435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0e9febb3-12b5-4fe0-908b-bfb88842900f admin admin] The token cache is being invalidate because project 0d062704975147a3b6b674b4e71f8e61 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:04.060740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0e9febb3-12b5-4fe0-908b-bfb88842900f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:04.061459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0e9febb3-12b5-4fe0-908b-bfb88842900f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:04.063147 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 952/1905] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:03 2022] DELETE /identity/v3/projects/0d062704975147a3b6b674b4e71f8e61 => generated 0 bytes in 616 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:04.069425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca494c7-faba-43ae-86a9-0b7b0b87fdbf None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:04.069633 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-407f310b-74f5-406c-b03a-a4c2818f38f6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '427969920afc46ee8adef66c927073be'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:04.069808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca494c7-faba-43ae-86a9-0b7b0b87fdbf None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:04.070105 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca494c7-faba-43ae-86a9-0b7b0b87fdbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:04.070527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-407f310b-74f5-406c-b03a-a4c2818f38f6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '427969920afc46ee8adef66c927073be'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:04.085495 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-407f310b-74f5-406c-b03a-a4c2818f38f6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '427969920afc46ee8adef66c927073be'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:04.114088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-fca494c7-faba-43ae-86a9-0b7b0b87fdbf None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:04.115327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-fca494c7-faba-43ae-86a9-0b7b0b87fdbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:04.184776 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 953/1906] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:04.193128 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d262b217-9cb3-493b-be24-61c56f81242f None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:04.193409 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d262b217-9cb3-493b-be24-61c56f81242f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:04.193679 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d262b217-9cb3-493b-be24-61c56f81242f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:04.214525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-d262b217-9cb3-493b-be24-61c56f81242f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:04.215152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-d262b217-9cb3-493b-be24-61c56f81242f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:04.233741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 954/1907] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:04.238987 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d262b217-9cb3-493b-be24-61c56f81242f None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:04.248491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d262b217-9cb3-493b-be24-61c56f81242f None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:04.253696 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52cc894-d0cf-4e01-8307-06f51b60cf0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:04.254364 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52cc894-d0cf-4e01-8307-06f51b60cf0a admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:04.254644 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52cc894-d0cf-4e01-8307-06f51b60cf0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:04.254914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52cc894-d0cf-4e01-8307-06f51b60cf0a admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:04.255655 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52cc894-d0cf-4e01-8307-06f51b60cf0a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:26:04.255940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52cc894-d0cf-4e01-8307-06f51b60cf0a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:04.257581 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52cc894-d0cf-4e01-8307-06f51b60cf0a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:04.266765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-407f310b-74f5-406c-b03a-a4c2818f38f6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '427969920afc46ee8adef66c927073be'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:04.291619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 954/1908] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:03 2022] DELETE /identity/v3/users/427969920afc46ee8adef66c927073be => generated 0 bytes in 715 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:04.291797 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 955/1909] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:26:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:26:04.294794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407f310b-74f5-406c-b03a-a4c2818f38f6 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:04.296354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-407f310b-74f5-406c-b03a-a4c2818f38f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:04.471903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407f310b-74f5-406c-b03a-a4c2818f38f6 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:04.477726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb24e3c-e052-40c4-a15c-6fd02c914c10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:04.478411 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb24e3c-e052-40c4-a15c-6fd02c914c10 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:04.478754 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb24e3c-e052-40c4-a15c-6fd02c914c10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:04.479117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb24e3c-e052-40c4-a15c-6fd02c914c10 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:04.479811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb24e3c-e052-40c4-a15c-6fd02c914c10 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:04.481982 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb24e3c-e052-40c4-a15c-6fd02c914c10 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:04.561079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:04.609870 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 955/1910] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:04 2022] POST /identity/v3/projects => generated 371 bytes in 316 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:04.614140 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52cc894-d0cf-4e01-8307-06f51b60cf0a admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:04.627004 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52cc894-d0cf-4e01-8307-06f51b60cf0a admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:04.738880 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfca80f-38de-42c3-8a5e-de545487f918 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:04.740558 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfca80f-38de-42c3-8a5e-de545487f918 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:04.740558 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfca80f-38de-42c3-8a5e-de545487f918 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:04.740992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfca80f-38de-42c3-8a5e-de545487f918 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:04.741867 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfca80f-38de-42c3-8a5e-de545487f918 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:04.745159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfca80f-38de-42c3-8a5e-de545487f918 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:05.009304 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 956/1911] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:04 2022] POST /identity/v3/users => generated 286 bytes in 396 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:05.013712 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb24e3c-e052-40c4-a15c-6fd02c914c10 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:05.024606 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb24e3c-e052-40c4-a15c-6fd02c914c10 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:05.029392 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e429ee-560a-47f7-92fd-af55733d6501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:05.029988 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e429ee-560a-47f7-92fd-af55733d6501 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:05.030616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e429ee-560a-47f7-92fd-af55733d6501 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:05.030901 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e429ee-560a-47f7-92fd-af55733d6501 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:05.031551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e429ee-560a-47f7-92fd-af55733d6501 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:05.033322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e429ee-560a-47f7-92fd-af55733d6501 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:05.045048 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 956/1912] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:05 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:05.050217 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfca80f-38de-42c3-8a5e-de545487f918 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:05.060750 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfca80f-38de-42c3-8a5e-de545487f918 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:05.065910 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c663929c-fbfd-4ff6-b0da-2472e26fad44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:05.066524 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c663929c-fbfd-4ff6-b0da-2472e26fad44 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:05.066810 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c663929c-fbfd-4ff6-b0da-2472e26fad44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:05.067154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c663929c-fbfd-4ff6-b0da-2472e26fad44 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:05.067800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c663929c-fbfd-4ff6-b0da-2472e26fad44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:05.069494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c663929c-fbfd-4ff6-b0da-2472e26fad44 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:05.084328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 957/1913] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:05 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:05.085087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e429ee-560a-47f7-92fd-af55733d6501 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:05.086822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c663929c-fbfd-4ff6-b0da-2472e26fad44 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:05.087151 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-59e429ee-560a-47f7-92fd-af55733d6501 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:05.102244 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c663929c-fbfd-4ff6-b0da-2472e26fad44 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:05.107224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85468ec3-2cfa-4072-9328-975ce207abfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:05.107986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85468ec3-2cfa-4072-9328-975ce207abfa admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:05.108293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85468ec3-2cfa-4072-9328-975ce207abfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:05.108592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85468ec3-2cfa-4072-9328-975ce207abfa admin admin] PATH_INFO: `/v3/projects/4973cc0ba2be421989db3ef0b7a1b174/users/cb82df06ea784f63bde89b26e2d76230/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:05.110147 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e429ee-560a-47f7-92fd-af55733d6501 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:05.119506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf623ae-31df-44b3-a26a-8fb8b841b9d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:05.120378 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf623ae-31df-44b3-a26a-8fb8b841b9d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:05.120378 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf623ae-31df-44b3-a26a-8fb8b841b9d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:05.120603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf623ae-31df-44b3-a26a-8fb8b841b9d4 admin admin] PATH_INFO: `/v3/projects/c25040ec09c64334a0016383be054b31` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:05.121786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf623ae-31df-44b3-a26a-8fb8b841b9d4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c25040ec09c64334a0016383be054b31)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:05.123723 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf623ae-31df-44b3-a26a-8fb8b841b9d4 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:05.128777 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85468ec3-2cfa-4072-9328-975ce207abfa admin admin] RBAC: Authorizing `identity:create_grant(project_id=4973cc0ba2be421989db3ef0b7a1b174, user_id=cb82df06ea784f63bde89b26e2d76230, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:05.133821 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85468ec3-2cfa-4072-9328-975ce207abfa admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:05.275909 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 958/1914] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:05 2022] PUT /identity/v3/projects/4973cc0ba2be421989db3ef0b7a1b174/users/cb82df06ea784f63bde89b26e2d76230/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:05.285700 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-eaf623ae-31df-44b3-a26a-8fb8b841b9d4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c25040ec09c64334a0016383be054b31', 'request_id': 'req-eaf623ae-31df-44b3-a26a-8fb8b841b9d4'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:05.290523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f0c02c-fb97-466c-860f-666a0edb4255 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:05.291079 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f0c02c-fb97-466c-860f-666a0edb4255 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:05.291500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f0c02c-fb97-466c-860f-666a0edb4255 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:05.292014 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-eaf623ae-31df-44b3-a26a-8fb8b841b9d4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c25040ec09c64334a0016383be054b31', 'request_id': 'req-eaf623ae-31df-44b3-a26a-8fb8b841b9d4'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:05.299660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-eaf623ae-31df-44b3-a26a-8fb8b841b9d4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c25040ec09c64334a0016383be054b31', 'request_id': 'req-eaf623ae-31df-44b3-a26a-8fb8b841b9d4'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:05.322973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-63f0c02c-fb97-466c-860f-666a0edb4255 None None] MFA Rules not processed for user `cb82df06ea784f63bde89b26e2d76230`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:05.324369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-63f0c02c-fb97-466c-860f-666a0edb4255 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:05.406650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 959/1915] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:05 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:05.420084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f0c02c-fb97-466c-860f-666a0edb4255 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:05.421679 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-63f0c02c-fb97-466c-860f-666a0edb4255 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:05.443018 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-eaf623ae-31df-44b3-a26a-8fb8b841b9d4 admin admin] The token cache is being invalidate because project c25040ec09c64334a0016383be054b31 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:05.443391 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-eaf623ae-31df-44b3-a26a-8fb8b841b9d4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:05.444477 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-eaf623ae-31df-44b3-a26a-8fb8b841b9d4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:05.446277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 957/1916] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:05 2022] DELETE /identity/v3/projects/c25040ec09c64334a0016383be054b31 => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:05.459174 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7da9f0-992b-40c9-b894-e530e2339a05 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:05.459542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7da9f0-992b-40c9-b894-e530e2339a05 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:05.459933 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7da9f0-992b-40c9-b894-e530e2339a05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:05.483623 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-9c7da9f0-992b-40c9-b894-e530e2339a05 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:05.484896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9c7da9f0-992b-40c9-b894-e530e2339a05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:05.497176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f0c02c-fb97-466c-860f-666a0edb4255 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:05.503406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f98004-c9e1-46dc-a802-bd8d5871c368 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:05.504039 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f98004-c9e1-46dc-a802-bd8d5871c368 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:05.504415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f98004-c9e1-46dc-a802-bd8d5871c368 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:05.504633 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f98004-c9e1-46dc-a802-bd8d5871c368 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:05.506574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-81f98004-c9e1-46dc-a802-bd8d5871c368 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:05.526351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f98004-c9e1-46dc-a802-bd8d5871c368 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:05.527973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f98004-c9e1-46dc-a802-bd8d5871c368 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:05.540477 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 960/1917] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:05 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:26:05.541645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f98004-c9e1-46dc-a802-bd8d5871c368 service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:05.543425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-81f98004-c9e1-46dc-a802-bd8d5871c368 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:05.593117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 958/1918] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:05.593117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7da9f0-992b-40c9-b894-e530e2339a05 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:05.603704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9c7da9f0-992b-40c9-b894-e530e2339a05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:05.617538 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7da9f0-992b-40c9-b894-e530e2339a05 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:05.640142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4411df39-43f6-4e3d-9dcb-a6795c75876d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:05.647874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4411df39-43f6-4e3d-9dcb-a6795c75876d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:05.647874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4411df39-43f6-4e3d-9dcb-a6795c75876d admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:05.647874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4411df39-43f6-4e3d-9dcb-a6795c75876d admin admin] PATH_INFO: `/v3/users/cb82df06ea784f63bde89b26e2d76230` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:05.647874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4411df39-43f6-4e3d-9dcb-a6795c75876d admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb82df06ea784f63bde89b26e2d76230)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:05.649866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4411df39-43f6-4e3d-9dcb-a6795c75876d admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:05.668717 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f98004-c9e1-46dc-a802-bd8d5871c368 service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:05.674059 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b59a8d2-cd93-4061-9a3d-ceed72f5250a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:05.674886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b59a8d2-cd93-4061-9a3d-ceed72f5250a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:05.675234 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b59a8d2-cd93-4061-9a3d-ceed72f5250a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:05.675572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b59a8d2-cd93-4061-9a3d-ceed72f5250a admin admin] PATH_INFO: `/v3/users/ef1b0ba55baa4805b8b21fcd86e801e1` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:05.677178 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b59a8d2-cd93-4061-9a3d-ceed72f5250a admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef1b0ba55baa4805b8b21fcd86e801e1)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:05.679357 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b59a8d2-cd93-4061-9a3d-ceed72f5250a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:05.986389 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2b59a8d2-cd93-4061-9a3d-ceed72f5250a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef1b0ba55baa4805b8b21fcd86e801e1'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:05.993491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2b59a8d2-cd93-4061-9a3d-ceed72f5250a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef1b0ba55baa4805b8b21fcd86e801e1'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:06.007272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2b59a8d2-cd93-4061-9a3d-ceed72f5250a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef1b0ba55baa4805b8b21fcd86e801e1'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:06.056298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4411df39-43f6-4e3d-9dcb-a6795c75876d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb82df06ea784f63bde89b26e2d76230'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:06.063493 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4411df39-43f6-4e3d-9dcb-a6795c75876d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb82df06ea784f63bde89b26e2d76230'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:06.064692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4411df39-43f6-4e3d-9dcb-a6795c75876d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb82df06ea784f63bde89b26e2d76230'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:06.081148 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4411df39-43f6-4e3d-9dcb-a6795c75876d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb82df06ea784f63bde89b26e2d76230'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:06.192089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2b59a8d2-cd93-4061-9a3d-ceed72f5250a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef1b0ba55baa4805b8b21fcd86e801e1'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:06.192949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2b59a8d2-cd93-4061-9a3d-ceed72f5250a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef1b0ba55baa4805b8b21fcd86e801e1'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:06.202839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 961/1919] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:05 2022] DELETE /identity/v3/users/ef1b0ba55baa4805b8b21fcd86e801e1 => generated 0 bytes in 662 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:06.204758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d698a0-4c16-47a8-9d66-0c6d146a4501 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:06.204758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d698a0-4c16-47a8-9d66-0c6d146a4501 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:06.204970 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d698a0-4c16-47a8-9d66-0c6d146a4501 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:06.227859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-75d698a0-4c16-47a8-9d66-0c6d146a4501 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:06.229228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-75d698a0-4c16-47a8-9d66-0c6d146a4501 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:06.251434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:06.307456 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 962/1920] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:06.308805 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcf17d9-67a0-4564-a280-74b73e71a795 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:06.309097 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcf17d9-67a0-4564-a280-74b73e71a795 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:06.309361 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcf17d9-67a0-4564-a280-74b73e71a795 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:06.329663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-cdcf17d9-67a0-4564-a280-74b73e71a795 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:06.331700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-cdcf17d9-67a0-4564-a280-74b73e71a795 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:06.344137 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 963/1921] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:06.344137 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcf17d9-67a0-4564-a280-74b73e71a795 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:06.365339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcf17d9-67a0-4564-a280-74b73e71a795 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:06.371279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4411df39-43f6-4e3d-9dcb-a6795c75876d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb82df06ea784f63bde89b26e2d76230'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:06.385946 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40155f35-a43d-4feb-87bf-fe8018db5937 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:06.386673 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40155f35-a43d-4feb-87bf-fe8018db5937 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:06.387074 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40155f35-a43d-4feb-87bf-fe8018db5937 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:06.387446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40155f35-a43d-4feb-87bf-fe8018db5937 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:06.387959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 959/1922] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:05 2022] DELETE /identity/v3/users/cb82df06ea784f63bde89b26e2d76230 => generated 0 bytes in 795 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:06.388834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40155f35-a43d-4feb-87bf-fe8018db5937 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:26:06.389585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40155f35-a43d-4feb-87bf-fe8018db5937 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:06.391380 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40155f35-a43d-4feb-87bf-fe8018db5937 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:06.392247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4411df39-43f6-4e3d-9dcb-a6795c75876d admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:06.394500 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-4411df39-43f6-4e3d-9dcb-a6795c75876d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:06.412954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 964/1923] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:26:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:26:06.414674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6355cf84-078a-43b9-885b-ba60ca67c927 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:06.414973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6355cf84-078a-43b9-885b-ba60ca67c927 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:06.415315 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6355cf84-078a-43b9-885b-ba60ca67c927 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:06.438926 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-6355cf84-078a-43b9-885b-ba60ca67c927 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:06.440901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6355cf84-078a-43b9-885b-ba60ca67c927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:06.512997 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4411df39-43f6-4e3d-9dcb-a6795c75876d admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:06.519536 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d883d240-d4f0-4f73-a2ed-abf6abdeaf7b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:06.535132 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d883d240-d4f0-4f73-a2ed-abf6abdeaf7b service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:06.536702 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 965/1924] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:06.539078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6355cf84-078a-43b9-885b-ba60ca67c927 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:06.540416 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d883d240-d4f0-4f73-a2ed-abf6abdeaf7b service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:06.541475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d883d240-d4f0-4f73-a2ed-abf6abdeaf7b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:06.543058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6355cf84-078a-43b9-885b-ba60ca67c927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:06.546911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d883d240-d4f0-4f73-a2ed-abf6abdeaf7b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:06.560904 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6355cf84-078a-43b9-885b-ba60ca67c927 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:06.571264 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d242934f-9365-4467-a6c8-4696d5923896 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:06.573274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d242934f-9365-4467-a6c8-4696d5923896 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:06.573679 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d242934f-9365-4467-a6c8-4696d5923896 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:06.574006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d242934f-9365-4467-a6c8-4696d5923896 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:06.574669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d242934f-9365-4467-a6c8-4696d5923896 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:06.576364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d883d240-d4f0-4f73-a2ed-abf6abdeaf7b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:06.579990 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d883d240-d4f0-4f73-a2ed-abf6abdeaf7b service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:06.581884 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d242934f-9365-4467-a6c8-4696d5923896 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:06.609893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 960/1925] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:06.615819 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d883d240-d4f0-4f73-a2ed-abf6abdeaf7b service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:06.639306 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d883d240-d4f0-4f73-a2ed-abf6abdeaf7b service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:06.646354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7b342d-fb87-48b6-a666-fc3cdd3942ba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:06.647535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7b342d-fb87-48b6-a666-fc3cdd3942ba service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:06.648035 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7b342d-fb87-48b6-a666-fc3cdd3942ba service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:06.648495 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7b342d-fb87-48b6-a666-fc3cdd3942ba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:06.686507 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7b342d-fb87-48b6-a666-fc3cdd3942ba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:06.689569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7b342d-fb87-48b6-a666-fc3cdd3942ba service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:06.716014 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 961/1926] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:06.873551 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 966/1927] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:06 2022] POST /identity/v3/projects => generated 383 bytes in 332 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:06.886102 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7b342d-fb87-48b6-a666-fc3cdd3942ba service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:06.906944 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7b342d-fb87-48b6-a666-fc3cdd3942ba service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:06.998073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d566bc5e-9063-4136-acdc-cdbfcbfe9bcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:06.998990 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d566bc5e-9063-4136-acdc-cdbfcbfe9bcc admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:06.999338 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d566bc5e-9063-4136-acdc-cdbfcbfe9bcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:06.999747 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d566bc5e-9063-4136-acdc-cdbfcbfe9bcc admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:07.000686 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d566bc5e-9063-4136-acdc-cdbfcbfe9bcc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:07.003000 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d566bc5e-9063-4136-acdc-cdbfcbfe9bcc admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:07.435003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 962/1928] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:06 2022] POST /identity/v3/users => generated 292 bytes in 549 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:07.442788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d242934f-9365-4467-a6c8-4696d5923896 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:07.454009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d242934f-9365-4467-a6c8-4696d5923896 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:07.459911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35d90b4-62ca-44c8-b4dd-521e5fa439f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:07.460583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35d90b4-62ca-44c8-b4dd-521e5fa439f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:07.460856 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35d90b4-62ca-44c8-b4dd-521e5fa439f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:07.461427 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35d90b4-62ca-44c8-b4dd-521e5fa439f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:07.462620 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35d90b4-62ca-44c8-b4dd-521e5fa439f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:07.463702 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35d90b4-62ca-44c8-b4dd-521e5fa439f6 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:07.479280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 967/1929] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:07 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:07.480649 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d566bc5e-9063-4136-acdc-cdbfcbfe9bcc admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:07.494414 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d566bc5e-9063-4136-acdc-cdbfcbfe9bcc admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:07.505093 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3497bb18-02db-4bb9-9a04-864aebbb4522 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:07.506164 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3497bb18-02db-4bb9-9a04-864aebbb4522 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:07.506450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3497bb18-02db-4bb9-9a04-864aebbb4522 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:07.506719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3497bb18-02db-4bb9-9a04-864aebbb4522 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:07.507735 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3497bb18-02db-4bb9-9a04-864aebbb4522 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:07.509484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3497bb18-02db-4bb9-9a04-864aebbb4522 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:07.524294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 963/1930] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:07 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:07.525717 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35d90b4-62ca-44c8-b4dd-521e5fa439f6 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:07.537544 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35d90b4-62ca-44c8-b4dd-521e5fa439f6 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:07.543224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a52a4ed-9b57-460d-a772-d387a1481124 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:07.544012 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a52a4ed-9b57-460d-a772-d387a1481124 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:07.544322 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a52a4ed-9b57-460d-a772-d387a1481124 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:07.545108 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a52a4ed-9b57-460d-a772-d387a1481124 admin admin] PATH_INFO: `/v3/projects/fe04655fd9114515bfbf8bec4450b48d/users/90dd2d2281df4daa96b105bf359b5297/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:07.565474 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a52a4ed-9b57-460d-a772-d387a1481124 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe04655fd9114515bfbf8bec4450b48d, user_id=90dd2d2281df4daa96b105bf359b5297, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:07.568316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a52a4ed-9b57-460d-a772-d387a1481124 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:07.761763 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:07.770319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 968/1931] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:07 2022] PUT /identity/v3/projects/fe04655fd9114515bfbf8bec4450b48d/users/90dd2d2281df4daa96b105bf359b5297/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:07.781778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce40370a-33de-484e-8a47-112e30936d53 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:07.782058 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce40370a-33de-484e-8a47-112e30936d53 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:07.782355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce40370a-33de-484e-8a47-112e30936d53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:07.827924 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-ce40370a-33de-484e-8a47-112e30936d53 None None] MFA Rules not processed for user `90dd2d2281df4daa96b105bf359b5297`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:07.829971 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ce40370a-33de-484e-8a47-112e30936d53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:07.919947 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 964/1932] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:07 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:07.925647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a52a4ed-9b57-460d-a772-d387a1481124 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:07.997618 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a52a4ed-9b57-460d-a772-d387a1481124 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:08.003460 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d1f300-9866-4625-a433-e5b603b9ed22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:08.004269 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d1f300-9866-4625-a433-e5b603b9ed22 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:08.004614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d1f300-9866-4625-a433-e5b603b9ed22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:08.004949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d1f300-9866-4625-a433-e5b603b9ed22 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:08.005587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d1f300-9866-4625-a433-e5b603b9ed22 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:08.008131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d1f300-9866-4625-a433-e5b603b9ed22 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:08.110559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce40370a-33de-484e-8a47-112e30936d53 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:08.111992 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ce40370a-33de-484e-8a47-112e30936d53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:08.157796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce40370a-33de-484e-8a47-112e30936d53 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:08.171673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:08.172387 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:08.172618 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:08.172893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a admin admin] PATH_INFO: `/v3/projects/4973cc0ba2be421989db3ef0b7a1b174` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:08.174177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a admin admin] RBAC: Authorizing `identity:delete_project(project_id=4973cc0ba2be421989db3ef0b7a1b174)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:08.176130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:08.262080 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 969/1933] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:07 2022] POST /identity/v3/projects => generated 383 bytes in 338 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:08.268357 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d1f300-9866-4625-a433-e5b603b9ed22 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:08.268357 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-65d1f300-9866-4625-a433-e5b603b9ed22 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:08.378985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d1f300-9866-4625-a433-e5b603b9ed22 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:08.384497 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff7aab8-a66c-4886-bbe3-d5a06dab9441 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:08.386304 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff7aab8-a66c-4886-bbe3-d5a06dab9441 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:08.387326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff7aab8-a66c-4886-bbe3-d5a06dab9441 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:08.387790 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff7aab8-a66c-4886-bbe3-d5a06dab9441 admin admin] PATH_INFO: `/v3/projects/063b0c2f3f7942cf95d5cd5ee6c509ef` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:08.389822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff7aab8-a66c-4886-bbe3-d5a06dab9441 admin admin] RBAC: Authorizing `identity:delete_project(project_id=063b0c2f3f7942cf95d5cd5ee6c509ef)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:08.392219 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff7aab8-a66c-4886-bbe3-d5a06dab9441 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:08.448093 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4973cc0ba2be421989db3ef0b7a1b174', 'request_id': 'req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:08.455134 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4973cc0ba2be421989db3ef0b7a1b174', 'request_id': 'req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:08.485121 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4973cc0ba2be421989db3ef0b7a1b174', 'request_id': 'req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:08.608861 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a admin admin] The token cache is being invalidate because project 4973cc0ba2be421989db3ef0b7a1b174 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:08.609157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:08.610619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:08.628386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 965/1934] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:08 2022] DELETE /identity/v3/projects/4973cc0ba2be421989db3ef0b7a1b174 => generated 0 bytes in 504 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:08.630236 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:08.643387 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:08.715403 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-eff7aab8-a66c-4886-bbe3-d5a06dab9441 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '063b0c2f3f7942cf95d5cd5ee6c509ef', 'request_id': 'req-eff7aab8-a66c-4886-bbe3-d5a06dab9441'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:08.811101 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-eff7aab8-a66c-4886-bbe3-d5a06dab9441 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '063b0c2f3f7942cf95d5cd5ee6c509ef', 'request_id': 'req-eff7aab8-a66c-4886-bbe3-d5a06dab9441'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:08.819723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-eff7aab8-a66c-4886-bbe3-d5a06dab9441 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '063b0c2f3f7942cf95d5cd5ee6c509ef', 'request_id': 'req-eff7aab8-a66c-4886-bbe3-d5a06dab9441'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:08.846291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-eff7aab8-a66c-4886-bbe3-d5a06dab9441 admin admin] The token cache is being invalidate because project 063b0c2f3f7942cf95d5cd5ee6c509ef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:08.846589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-eff7aab8-a66c-4886-bbe3-d5a06dab9441 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:08.847310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-eff7aab8-a66c-4886-bbe3-d5a06dab9441 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:08.849387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 970/1935] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:08 2022] DELETE /identity/v3/projects/063b0c2f3f7942cf95d5cd5ee6c509ef => generated 0 bytes in 587 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:08.860296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537e4e3c-2def-4381-a305-b0bdf4ae33a1 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:08.860296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537e4e3c-2def-4381-a305-b0bdf4ae33a1 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:08.860296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537e4e3c-2def-4381-a305-b0bdf4ae33a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:08.870027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dffe220-b0ee-4426-a3f5-fb5ceeaaed2a admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:08.879451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-537e4e3c-2def-4381-a305-b0bdf4ae33a1 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:08.881174 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-537e4e3c-2def-4381-a305-b0bdf4ae33a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:08.948210 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e899085f-16ea-4ea0-a44f-33451b21a668 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:08.948969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e899085f-16ea-4ea0-a44f-33451b21a668 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:08.949351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e899085f-16ea-4ea0-a44f-33451b21a668 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:08.949782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e899085f-16ea-4ea0-a44f-33451b21a668 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:08.950557 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e899085f-16ea-4ea0-a44f-33451b21a668 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:08.952762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e899085f-16ea-4ea0-a44f-33451b21a668 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:08.954267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 971/1936] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:08.962260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7a379c-0d92-4d23-917c-bb0c3f9f9fee None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:08.962783 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7a379c-0d92-4d23-917c-bb0c3f9f9fee None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:08.963108 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7a379c-0d92-4d23-917c-bb0c3f9f9fee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:08.985749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-5e7a379c-0d92-4d23-917c-bb0c3f9f9fee None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:08.987679 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-5e7a379c-0d92-4d23-917c-bb0c3f9f9fee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:08.998581 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 972/1937] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:09.004668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7a379c-0d92-4d23-917c-bb0c3f9f9fee None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:09.014766 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7a379c-0d92-4d23-917c-bb0c3f9f9fee None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:09.020545 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d310c659-842f-4523-97a7-54ce2e223552 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:09.021246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d310c659-842f-4523-97a7-54ce2e223552 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:09.021526 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d310c659-842f-4523-97a7-54ce2e223552 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:09.021840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d310c659-842f-4523-97a7-54ce2e223552 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:09.022508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d310c659-842f-4523-97a7-54ce2e223552 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:26:09.022811 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d310c659-842f-4523-97a7-54ce2e223552 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:09.024509 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d310c659-842f-4523-97a7-54ce2e223552 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:09.041866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 973/1938] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:26:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:26:09.047480 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d310c659-842f-4523-97a7-54ce2e223552 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:09.058279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d310c659-842f-4523-97a7-54ce2e223552 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:09.065094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f4dd93-ae1a-438d-aaa4-ad92e2421a82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:09.066315 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f4dd93-ae1a-438d-aaa4-ad92e2421a82 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:09.066981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f4dd93-ae1a-438d-aaa4-ad92e2421a82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:09.068450 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f4dd93-ae1a-438d-aaa4-ad92e2421a82 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:09.070504 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f4dd93-ae1a-438d-aaa4-ad92e2421a82 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:09.078231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f4dd93-ae1a-438d-aaa4-ad92e2421a82 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:09.078621 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 966/1939] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:08 2022] POST /identity/v3/users => generated 298 bytes in 449 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:09.085123 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e899085f-16ea-4ea0-a44f-33451b21a668 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:09.087006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e899085f-16ea-4ea0-a44f-33451b21a668 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:09.101883 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e899085f-16ea-4ea0-a44f-33451b21a668 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:09.108041 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ed17c2-d44d-4fd3-94e3-b4a7b5a6348f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:09.108872 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ed17c2-d44d-4fd3-94e3-b4a7b5a6348f admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:09.109247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ed17c2-d44d-4fd3-94e3-b4a7b5a6348f admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:09.109799 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ed17c2-d44d-4fd3-94e3-b4a7b5a6348f admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:09.111001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ed17c2-d44d-4fd3-94e3-b4a7b5a6348f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:09.113779 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ed17c2-d44d-4fd3-94e3-b4a7b5a6348f admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:09.133732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 967/1940] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:09 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:09.136902 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ed17c2-d44d-4fd3-94e3-b4a7b5a6348f admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:09.149289 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ed17c2-d44d-4fd3-94e3-b4a7b5a6348f admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:09.155921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0fecb7-69e3-423d-b14c-80ce1a144d45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:09.156737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0fecb7-69e3-423d-b14c-80ce1a144d45 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:09.157087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0fecb7-69e3-423d-b14c-80ce1a144d45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:09.157543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0fecb7-69e3-423d-b14c-80ce1a144d45 admin admin] PATH_INFO: `/v3/projects/29ca46f1519348d9b356f7e7e0aa0947/users/2e54bc1953244da389c2de3a00ab5f73/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:09.180369 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0fecb7-69e3-423d-b14c-80ce1a144d45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=29ca46f1519348d9b356f7e7e0aa0947, user_id=2e54bc1953244da389c2de3a00ab5f73, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:09.183371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0fecb7-69e3-423d-b14c-80ce1a144d45 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:09.216908 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:09.285478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 974/1941] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:09 2022] POST /identity/v3/projects => generated 369 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:09.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f4dd93-ae1a-438d-aaa4-ad92e2421a82 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:09.321637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 968/1942] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:09 2022] PUT /identity/v3/projects/29ca46f1519348d9b356f7e7e0aa0947/users/2e54bc1953244da389c2de3a00ab5f73/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:09.331465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0fecb7-69e3-423d-b14c-80ce1a144d45 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:09.412663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f4dd93-ae1a-438d-aaa4-ad92e2421a82 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:09.459524 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0fecb7-69e3-423d-b14c-80ce1a144d45 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:09.467541 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb848f5f-27d8-4641-bef4-4b954c3babc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:09.468681 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb848f5f-27d8-4641-bef4-4b954c3babc0 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:09.477361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb848f5f-27d8-4641-bef4-4b954c3babc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:09.477361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb848f5f-27d8-4641-bef4-4b954c3babc0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:09.477361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb848f5f-27d8-4641-bef4-4b954c3babc0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:09.477361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb848f5f-27d8-4641-bef4-4b954c3babc0 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:09.489739 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41417ade-5e38-4fff-ad5d-8a86edb49080 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:09.493737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41417ade-5e38-4fff-ad5d-8a86edb49080 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:09.493737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41417ade-5e38-4fff-ad5d-8a86edb49080 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:09.493737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41417ade-5e38-4fff-ad5d-8a86edb49080 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:09.493737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41417ade-5e38-4fff-ad5d-8a86edb49080 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:09.495169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 969/1943] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:09 2022] GET /identity/v3/roles => generated 1420 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:09.495632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41417ade-5e38-4fff-ad5d-8a86edb49080 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:09.496564 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb848f5f-27d8-4641-bef4-4b954c3babc0 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:09.516135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb848f5f-27d8-4641-bef4-4b954c3babc0 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:09.521866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5797d27-52f3-43fc-b2f3-49ee4f11f298 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:09.522552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5797d27-52f3-43fc-b2f3-49ee4f11f298 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:09.522830 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5797d27-52f3-43fc-b2f3-49ee4f11f298 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:09.523140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5797d27-52f3-43fc-b2f3-49ee4f11f298 admin admin] PATH_INFO: `/v3/projects/29ca46f1519348d9b356f7e7e0aa0947/users/2e54bc1953244da389c2de3a00ab5f73/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:09.530309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5797d27-52f3-43fc-b2f3-49ee4f11f298 admin admin] RBAC: Authorizing `identity:create_grant(project_id=29ca46f1519348d9b356f7e7e0aa0947, user_id=2e54bc1953244da389c2de3a00ab5f73, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:09.533899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5797d27-52f3-43fc-b2f3-49ee4f11f298 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:09.555285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 970/1944] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:09 2022] PUT /identity/v3/projects/29ca46f1519348d9b356f7e7e0aa0947/users/2e54bc1953244da389c2de3a00ab5f73/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:09.566224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3a24b8-b4c8-460e-82c1-65a3c8c4f7bf None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:09.566605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3a24b8-b4c8-460e-82c1-65a3c8c4f7bf None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:09.566890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3a24b8-b4c8-460e-82c1-65a3c8c4f7bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:09.618240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-7d3a24b8-b4c8-460e-82c1-65a3c8c4f7bf None None] MFA Rules not processed for user `2e54bc1953244da389c2de3a00ab5f73`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:09.620189 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7d3a24b8-b4c8-460e-82c1-65a3c8c4f7bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:09.621545 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 975/1945] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:09 2022] POST /identity/v3/users => generated 285 bytes in 333 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:09.629253 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41417ade-5e38-4fff-ad5d-8a86edb49080 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:09.652902 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41417ade-5e38-4fff-ad5d-8a86edb49080 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:09.733815 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 971/1946] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:09 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:09.741256 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3a24b8-b4c8-460e-82c1-65a3c8c4f7bf None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:09.742446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfaeeab-864e-4a7c-b625-143b6931b4cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:09.743017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7d3a24b8-b4c8-460e-82c1-65a3c8c4f7bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:09.743564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfaeeab-864e-4a7c-b625-143b6931b4cb admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:09.743985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfaeeab-864e-4a7c-b625-143b6931b4cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:09.744435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfaeeab-864e-4a7c-b625-143b6931b4cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:09.745231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfaeeab-864e-4a7c-b625-143b6931b4cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:09.747256 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfaeeab-864e-4a7c-b625-143b6931b4cb admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:09.767344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 976/1947] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:09 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:09.770146 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfaeeab-864e-4a7c-b625-143b6931b4cb admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:09.785655 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfaeeab-864e-4a7c-b625-143b6931b4cb admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:09.792411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014f849a-bb8e-462b-a057-f2ef22802981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:09.793526 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014f849a-bb8e-462b-a057-f2ef22802981 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:09.793950 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014f849a-bb8e-462b-a057-f2ef22802981 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:09.794362 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014f849a-bb8e-462b-a057-f2ef22802981 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:09.795309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014f849a-bb8e-462b-a057-f2ef22802981 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:09.797999 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014f849a-bb8e-462b-a057-f2ef22802981 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:09.816278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 977/1948] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:09 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:09.818423 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014f849a-bb8e-462b-a057-f2ef22802981 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:09.845244 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3a24b8-b4c8-460e-82c1-65a3c8c4f7bf None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:09.849967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014f849a-bb8e-462b-a057-f2ef22802981 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:09.850341 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2683e0a5-8994-44df-b76c-d90eb790a1d3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:09.852956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2683e0a5-8994-44df-b76c-d90eb790a1d3 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:09.853558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2683e0a5-8994-44df-b76c-d90eb790a1d3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:09.854124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2683e0a5-8994-44df-b76c-d90eb790a1d3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:09.868320 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2da3f8f-60bc-4088-a540-7e3bead9459e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:09.869513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2da3f8f-60bc-4088-a540-7e3bead9459e admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:09.869935 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2da3f8f-60bc-4088-a540-7e3bead9459e admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:09.871405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2da3f8f-60bc-4088-a540-7e3bead9459e admin admin] PATH_INFO: `/v3/projects/8ddaa0706bd54806973c06cb1dca3ed6/users/04456917f3c24a5dbb50e3a3eb187b10/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:09.880079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2683e0a5-8994-44df-b76c-d90eb790a1d3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:09.882486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2683e0a5-8994-44df-b76c-d90eb790a1d3 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:09.917088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2da3f8f-60bc-4088-a540-7e3bead9459e admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ddaa0706bd54806973c06cb1dca3ed6, user_id=04456917f3c24a5dbb50e3a3eb187b10, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:09.920726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 972/1949] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:09 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:09.938786 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2da3f8f-60bc-4088-a540-7e3bead9459e admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:10.094395 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 978/1950] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:09 2022] PUT /identity/v3/projects/8ddaa0706bd54806973c06cb1dca3ed6/users/04456917f3c24a5dbb50e3a3eb187b10/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:10.107365 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d21732-4af7-4be2-9a15-67f5568d56b7 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:10.107672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d21732-4af7-4be2-9a15-67f5568d56b7 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:10.108272 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d21732-4af7-4be2-9a15-67f5568d56b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:10.114393 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2da3f8f-60bc-4088-a540-7e3bead9459e admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:10.173616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-68d21732-4af7-4be2-9a15-67f5568d56b7 None None] MFA Rules not processed for user `04456917f3c24a5dbb50e3a3eb187b10`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:10.175203 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-68d21732-4af7-4be2-9a15-67f5568d56b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:10.227083 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2da3f8f-60bc-4088-a540-7e3bead9459e admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:10.232040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437da339-1d80-4102-9893-2a5af2fb8a98 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:10.233572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437da339-1d80-4102-9893-2a5af2fb8a98 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:10.233954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437da339-1d80-4102-9893-2a5af2fb8a98 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:10.234336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437da339-1d80-4102-9893-2a5af2fb8a98 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:10.236724 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-437da339-1d80-4102-9893-2a5af2fb8a98 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:10.283717 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 973/1951] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:10 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:10.299959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d21732-4af7-4be2-9a15-67f5568d56b7 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:10.302757 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-68d21732-4af7-4be2-9a15-67f5568d56b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:10.321979 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:10.360343 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437da339-1d80-4102-9893-2a5af2fb8a98 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:10.363386 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437da339-1d80-4102-9893-2a5af2fb8a98 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:10.371592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 979/1952] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:10 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 258 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:10.421187 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d21732-4af7-4be2-9a15-67f5568d56b7 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:10.426333 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb2731b-9aa8-43f9-aa30-7205532cced1 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6365b2328f34ab9b1b73ececbf5cc30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:10.427148 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb2731b-9aa8-43f9-aa30-7205532cced1 service glance] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:10.427528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb2731b-9aa8-43f9-aa30-7205532cced1 service glance] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:10.427907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb2731b-9aa8-43f9-aa30-7205532cced1 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:10.438607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb2731b-9aa8-43f9-aa30-7205532cced1 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:10.441395 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb2731b-9aa8-43f9-aa30-7205532cced1 service glance] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:10.464039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 974/1953] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:10 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:10.703740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437da339-1d80-4102-9893-2a5af2fb8a98 service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:10.705817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-437da339-1d80-4102-9893-2a5af2fb8a98 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:10.794103 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:10.842896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437da339-1d80-4102-9893-2a5af2fb8a98 service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:10.850326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2643ad58-1074-42b0-a7b2-2611994ce649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:10.850994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2643ad58-1074-42b0-a7b2-2611994ce649 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:10.851275 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2643ad58-1074-42b0-a7b2-2611994ce649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:10.851550 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2643ad58-1074-42b0-a7b2-2611994ce649 admin admin] PATH_INFO: `/v3/users/f855040e6fd74026854fb997d888df17` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:10.852862 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2643ad58-1074-42b0-a7b2-2611994ce649 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f855040e6fd74026854fb997d888df17)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:10.855443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2643ad58-1074-42b0-a7b2-2611994ce649 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:11.084414 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb2731b-9aa8-43f9-aa30-7205532cced1 service glance] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:11.096924 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb2731b-9aa8-43f9-aa30-7205532cced1 service glance] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:11.100338 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2643ad58-1074-42b0-a7b2-2611994ce649 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f855040e6fd74026854fb997d888df17'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:11.102250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c235312-b308-4d9b-8a8b-95244fe3f363 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:11.103026 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c235312-b308-4d9b-8a8b-95244fe3f363 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:11.103536 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c235312-b308-4d9b-8a8b-95244fe3f363 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:11.103904 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c235312-b308-4d9b-8a8b-95244fe3f363 admin admin] PATH_INFO: `/v3/users/90dd2d2281df4daa96b105bf359b5297` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:11.105213 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c235312-b308-4d9b-8a8b-95244fe3f363 admin admin] RBAC: Authorizing `identity:delete_user(user_id=90dd2d2281df4daa96b105bf359b5297)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:11.107932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c235312-b308-4d9b-8a8b-95244fe3f363 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:11.108406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2643ad58-1074-42b0-a7b2-2611994ce649 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f855040e6fd74026854fb997d888df17'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:11.122602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2643ad58-1074-42b0-a7b2-2611994ce649 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f855040e6fd74026854fb997d888df17'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:11.241572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2643ad58-1074-42b0-a7b2-2611994ce649 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f855040e6fd74026854fb997d888df17'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:11.242424 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2643ad58-1074-42b0-a7b2-2611994ce649 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f855040e6fd74026854fb997d888df17'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:11.250457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 980/1954] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:10 2022] DELETE /identity/v3/users/f855040e6fd74026854fb997d888df17 => generated 0 bytes in 547 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:11.255247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70339b1c-4df3-4995-a80a-91a1c9c314b4 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:11.255471 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70339b1c-4df3-4995-a80a-91a1c9c314b4 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:11.255742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70339b1c-4df3-4995-a80a-91a1c9c314b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:11.277765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-70339b1c-4df3-4995-a80a-91a1c9c314b4 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:11.279065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-70339b1c-4df3-4995-a80a-91a1c9c314b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:11.346533 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7c235312-b308-4d9b-8a8b-95244fe3f363 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '90dd2d2281df4daa96b105bf359b5297'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:11.369519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 981/1955] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:11.369611 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7c235312-b308-4d9b-8a8b-95244fe3f363 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '90dd2d2281df4daa96b105bf359b5297'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:11.369611 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7c235312-b308-4d9b-8a8b-95244fe3f363 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '90dd2d2281df4daa96b105bf359b5297'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:11.377948 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70339b1c-4df3-4995-a80a-91a1c9c314b4 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:11.379412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-70339b1c-4df3-4995-a80a-91a1c9c314b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:11.383380 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7c235312-b308-4d9b-8a8b-95244fe3f363 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '90dd2d2281df4daa96b105bf359b5297'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:11.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70339b1c-4df3-4995-a80a-91a1c9c314b4 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:11.442857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513ed827-3e78-4a46-83f5-e7b664ccb794 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:11.443471 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513ed827-3e78-4a46-83f5-e7b664ccb794 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:11.443747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513ed827-3e78-4a46-83f5-e7b664ccb794 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:11.444031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513ed827-3e78-4a46-83f5-e7b664ccb794 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:11.453682 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513ed827-3e78-4a46-83f5-e7b664ccb794 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:11.455937 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513ed827-3e78-4a46-83f5-e7b664ccb794 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:11.466236 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 982/1956] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:11.485315 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7c235312-b308-4d9b-8a8b-95244fe3f363 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '90dd2d2281df4daa96b105bf359b5297'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:11.496643 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 975/1957] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:11 2022] DELETE /identity/v3/users/90dd2d2281df4daa96b105bf359b5297 => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:11.505846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0494ea-4c4b-49a7-9fe0-e40edbd912fe None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:11.506315 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0494ea-4c4b-49a7-9fe0-e40edbd912fe None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:11.506758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0494ea-4c4b-49a7-9fe0-e40edbd912fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:11.543248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-dd0494ea-4c4b-49a7-9fe0-e40edbd912fe None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:11.544784 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-dd0494ea-4c4b-49a7-9fe0-e40edbd912fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:11.616962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 983/1958] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:11.628026 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c235312-b308-4d9b-8a8b-95244fe3f363 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:11.629557 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7c235312-b308-4d9b-8a8b-95244fe3f363 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:11.707416 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c235312-b308-4d9b-8a8b-95244fe3f363 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:11.711539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296aadd6-0d20-413e-b241-6bc2c24ff466 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:11.712182 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296aadd6-0d20-413e-b241-6bc2c24ff466 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:11.712461 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296aadd6-0d20-413e-b241-6bc2c24ff466 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:11.712726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296aadd6-0d20-413e-b241-6bc2c24ff466 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:11.724400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296aadd6-0d20-413e-b241-6bc2c24ff466 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:11.725859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296aadd6-0d20-413e-b241-6bc2c24ff466 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:11.735789 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 976/1959] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:12.093037 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0494ea-4c4b-49a7-9fe0-e40edbd912fe None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:12.094567 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-dd0494ea-4c4b-49a7-9fe0-e40edbd912fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:12.112949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0494ea-4c4b-49a7-9fe0-e40edbd912fe None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:12.124231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:12.129883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:12.129883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:12.129883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43 admin admin] PATH_INFO: `/v3/projects/02ec9f9599c9451fabc232e5cb0933b1` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:12.129883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43 admin admin] RBAC: Authorizing `identity:delete_project(project_id=02ec9f9599c9451fabc232e5cb0933b1)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:12.131887 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:12.354579 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296aadd6-0d20-413e-b241-6bc2c24ff466 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:12.355962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-296aadd6-0d20-413e-b241-6bc2c24ff466 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:12.369857 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296aadd6-0d20-413e-b241-6bc2c24ff466 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:12.380529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02ec9f9599c9451fabc232e5cb0933b1', 'request_id': 'req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:12.435863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:12.456484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b28645-a9f1-4b80-8712-91eb56526a59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:12.457459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b28645-a9f1-4b80-8712-91eb56526a59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:12.457561 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b28645-a9f1-4b80-8712-91eb56526a59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:12.457809 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b28645-a9f1-4b80-8712-91eb56526a59 admin admin] PATH_INFO: `/v3/projects/fe04655fd9114515bfbf8bec4450b48d` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:12.459431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b28645-a9f1-4b80-8712-91eb56526a59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fe04655fd9114515bfbf8bec4450b48d)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:12.461824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b28645-a9f1-4b80-8712-91eb56526a59 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:12.473060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02ec9f9599c9451fabc232e5cb0933b1', 'request_id': 'req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:12.481609 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02ec9f9599c9451fabc232e5cb0933b1', 'request_id': 'req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:12.495316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43 admin admin] The token cache is being invalidate because project 02ec9f9599c9451fabc232e5cb0933b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:12.495799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:12.500678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:12.505196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 984/1960] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:12 2022] DELETE /identity/v3/projects/02ec9f9599c9451fabc232e5cb0933b1 => generated 0 bytes in 411 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:12.505362 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:12.507121 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:12.508712 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:12.524443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44a6d26-4bb2-4c59-b993-e3da9bcdab43 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:12.529181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa97c91f-1793-4bc8-9482-284cc4f95c40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:12.529822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa97c91f-1793-4bc8-9482-284cc4f95c40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:12.530095 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa97c91f-1793-4bc8-9482-284cc4f95c40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:12.530363 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa97c91f-1793-4bc8-9482-284cc4f95c40 admin admin] PATH_INFO: `/v3/users/ba7b6a5ee9b84d4ea25b1cfc860ca949` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:12.531571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa97c91f-1793-4bc8-9482-284cc4f95c40 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba7b6a5ee9b84d4ea25b1cfc860ca949)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:12.533421 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa97c91f-1793-4bc8-9482-284cc4f95c40 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:12.639398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e7b28645-a9f1-4b80-8712-91eb56526a59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fe04655fd9114515bfbf8bec4450b48d', 'request_id': 'req-e7b28645-a9f1-4b80-8712-91eb56526a59'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:12.646892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e7b28645-a9f1-4b80-8712-91eb56526a59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fe04655fd9114515bfbf8bec4450b48d', 'request_id': 'req-e7b28645-a9f1-4b80-8712-91eb56526a59'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:12.654489 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e7b28645-a9f1-4b80-8712-91eb56526a59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fe04655fd9114515bfbf8bec4450b48d', 'request_id': 'req-e7b28645-a9f1-4b80-8712-91eb56526a59'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:12.760149 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e7b28645-a9f1-4b80-8712-91eb56526a59 admin admin] The token cache is being invalidate because project fe04655fd9114515bfbf8bec4450b48d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:12.760448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e7b28645-a9f1-4b80-8712-91eb56526a59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:12.761212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e7b28645-a9f1-4b80-8712-91eb56526a59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:12.762863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 977/1961] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:12 2022] DELETE /identity/v3/projects/fe04655fd9114515bfbf8bec4450b48d => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:12.767501 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b28645-a9f1-4b80-8712-91eb56526a59 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:12.769391 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e7b28645-a9f1-4b80-8712-91eb56526a59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:12.771060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-fa97c91f-1793-4bc8-9482-284cc4f95c40 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba7b6a5ee9b84d4ea25b1cfc860ca949'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:12.779100 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-fa97c91f-1793-4bc8-9482-284cc4f95c40 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba7b6a5ee9b84d4ea25b1cfc860ca949'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:12.791738 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-fa97c91f-1793-4bc8-9482-284cc4f95c40 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba7b6a5ee9b84d4ea25b1cfc860ca949'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:12.842448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b28645-a9f1-4b80-8712-91eb56526a59 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:12.847345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d311909-740d-4ecc-baa1-b0116f2a8966 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:12.848039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d311909-740d-4ecc-baa1-b0116f2a8966 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:12.848306 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d311909-740d-4ecc-baa1-b0116f2a8966 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:12.848591 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d311909-740d-4ecc-baa1-b0116f2a8966 admin admin] PATH_INFO: `/v3/users/2e54bc1953244da389c2de3a00ab5f73` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:12.849744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d311909-740d-4ecc-baa1-b0116f2a8966 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e54bc1953244da389c2de3a00ab5f73)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:12.851746 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d311909-740d-4ecc-baa1-b0116f2a8966 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:12.898546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-fa97c91f-1793-4bc8-9482-284cc4f95c40 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba7b6a5ee9b84d4ea25b1cfc860ca949'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:12.899165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-fa97c91f-1793-4bc8-9482-284cc4f95c40 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba7b6a5ee9b84d4ea25b1cfc860ca949'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:12.905665 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 985/1962] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:12 2022] DELETE /identity/v3/users/ba7b6a5ee9b84d4ea25b1cfc860ca949 => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:13.172780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5d311909-740d-4ecc-baa1-b0116f2a8966 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e54bc1953244da389c2de3a00ab5f73'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:13.192424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5d311909-740d-4ecc-baa1-b0116f2a8966 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e54bc1953244da389c2de3a00ab5f73'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:13.193247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5d311909-740d-4ecc-baa1-b0116f2a8966 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e54bc1953244da389c2de3a00ab5f73'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:13.206966 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5d311909-740d-4ecc-baa1-b0116f2a8966 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e54bc1953244da389c2de3a00ab5f73'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:13.318884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5d311909-740d-4ecc-baa1-b0116f2a8966 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e54bc1953244da389c2de3a00ab5f73'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:13.325582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 978/1963] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:12 2022] DELETE /identity/v3/users/2e54bc1953244da389c2de3a00ab5f73 => generated 0 bytes in 559 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:13.422072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa97c91f-1793-4bc8-9482-284cc4f95c40 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:13.423579 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-fa97c91f-1793-4bc8-9482-284cc4f95c40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:13.502201 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa97c91f-1793-4bc8-9482-284cc4f95c40 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:13.560986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d041a7e-88a3-430c-9d25-69b1a1379e7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:13.561629 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d041a7e-88a3-430c-9d25-69b1a1379e7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:13.561962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d041a7e-88a3-430c-9d25-69b1a1379e7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:13.562260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d041a7e-88a3-430c-9d25-69b1a1379e7c admin admin] PATH_INFO: `/v3/projects/936fb875ffe442cd82df135b92abb639` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:13.563583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d041a7e-88a3-430c-9d25-69b1a1379e7c admin admin] RBAC: Authorizing `identity:delete_project(project_id=936fb875ffe442cd82df135b92abb639)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:13.565651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d041a7e-88a3-430c-9d25-69b1a1379e7c admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:13.595210 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:INSERT=5,keystone:SELECT=91,keystone:DELETE=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:13.747919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3d041a7e-88a3-430c-9d25-69b1a1379e7c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '936fb875ffe442cd82df135b92abb639', 'request_id': 'req-3d041a7e-88a3-430c-9d25-69b1a1379e7c'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:13.844838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3d041a7e-88a3-430c-9d25-69b1a1379e7c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '936fb875ffe442cd82df135b92abb639', 'request_id': 'req-3d041a7e-88a3-430c-9d25-69b1a1379e7c'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:13.851267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3d041a7e-88a3-430c-9d25-69b1a1379e7c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '936fb875ffe442cd82df135b92abb639', 'request_id': 'req-3d041a7e-88a3-430c-9d25-69b1a1379e7c'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:13.864694 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3d041a7e-88a3-430c-9d25-69b1a1379e7c admin admin] The token cache is being invalidate because project 936fb875ffe442cd82df135b92abb639 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:13.864978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3d041a7e-88a3-430c-9d25-69b1a1379e7c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:13.865638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3d041a7e-88a3-430c-9d25-69b1a1379e7c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:13.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 986/1964] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:13 2022] DELETE /identity/v3/projects/936fb875ffe442cd82df135b92abb639 => generated 0 bytes in 446 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:13.872181 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58603bc-0b85-4a9b-960f-7f6c39026ca1 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:13.872636 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58603bc-0b85-4a9b-960f-7f6c39026ca1 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:13.872725 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58603bc-0b85-4a9b-960f-7f6c39026ca1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:13.892274 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f58603bc-0b85-4a9b-960f-7f6c39026ca1 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:13.894089 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f58603bc-0b85-4a9b-960f-7f6c39026ca1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:13.961148 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 979/1965] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:13.969491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109436b0-4dbe-4526-b8b4-5fcb4fa4387f None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:13.969770 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109436b0-4dbe-4526-b8b4-5fcb4fa4387f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:13.974390 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109436b0-4dbe-4526-b8b4-5fcb4fa4387f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:13.990256 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-109436b0-4dbe-4526-b8b4-5fcb4fa4387f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:13.991491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-109436b0-4dbe-4526-b8b4-5fcb4fa4387f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:14.000268 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 987/1966] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:14.007330 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58603bc-0b85-4a9b-960f-7f6c39026ca1 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:14.009022 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109436b0-4dbe-4526-b8b4-5fcb4fa4387f None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:14.009196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f58603bc-0b85-4a9b-960f-7f6c39026ca1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:14.023310 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58603bc-0b85-4a9b-960f-7f6c39026ca1 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:14.026238 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:14.028287 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109436b0-4dbe-4526-b8b4-5fcb4fa4387f None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:14.031358 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20ad876-aebe-4d81-bc51-a669f651328b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:14.032488 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20ad876-aebe-4d81-bc51-a669f651328b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.032765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20ad876-aebe-4d81-bc51-a669f651328b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.033032 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20ad876-aebe-4d81-bc51-a669f651328b admin admin] PATH_INFO: `/v3/projects/29ca46f1519348d9b356f7e7e0aa0947` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.033343 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c180ee38-eb4e-493a-adce-3b6167932651 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:14.033990 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c180ee38-eb4e-493a-adce-3b6167932651 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.034225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20ad876-aebe-4d81-bc51-a669f651328b admin admin] RBAC: Authorizing `identity:delete_project(project_id=29ca46f1519348d9b356f7e7e0aa0947)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:14.034334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c180ee38-eb4e-493a-adce-3b6167932651 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.034525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c180ee38-eb4e-493a-adce-3b6167932651 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.035337 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c180ee38-eb4e-493a-adce-3b6167932651 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:26:14.035640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c180ee38-eb4e-493a-adce-3b6167932651 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:14.039817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20ad876-aebe-4d81-bc51-a669f651328b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:14.040055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c180ee38-eb4e-493a-adce-3b6167932651 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:14.050560 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 988/1967] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:26:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:26:14.054824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c180ee38-eb4e-493a-adce-3b6167932651 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:14.066396 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c180ee38-eb4e-493a-adce-3b6167932651 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:14.072157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b530ffe-e928-41d2-b3d8-4268a0172145 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:14.072985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b530ffe-e928-41d2-b3d8-4268a0172145 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.074435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b530ffe-e928-41d2-b3d8-4268a0172145 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.074435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b530ffe-e928-41d2-b3d8-4268a0172145 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.074911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b530ffe-e928-41d2-b3d8-4268a0172145 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:14.076943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b530ffe-e928-41d2-b3d8-4268a0172145 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:14.179791 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 989/1968] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:14 2022] POST /identity/v3/projects => generated 385 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:14.184367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b530ffe-e928-41d2-b3d8-4268a0172145 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:14.193922 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e20ad876-aebe-4d81-bc51-a669f651328b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '29ca46f1519348d9b356f7e7e0aa0947', 'request_id': 'req-e20ad876-aebe-4d81-bc51-a669f651328b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:14.200201 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e20ad876-aebe-4d81-bc51-a669f651328b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '29ca46f1519348d9b356f7e7e0aa0947', 'request_id': 'req-e20ad876-aebe-4d81-bc51-a669f651328b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:14.207855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e20ad876-aebe-4d81-bc51-a669f651328b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '29ca46f1519348d9b356f7e7e0aa0947', 'request_id': 'req-e20ad876-aebe-4d81-bc51-a669f651328b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:14.255696 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b530ffe-e928-41d2-b3d8-4268a0172145 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:14.260503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9879be32-ed3e-46e9-9d1e-b70453525fee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:14.261118 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9879be32-ed3e-46e9-9d1e-b70453525fee admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.261799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9879be32-ed3e-46e9-9d1e-b70453525fee admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.261799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9879be32-ed3e-46e9-9d1e-b70453525fee admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.262331 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9879be32-ed3e-46e9-9d1e-b70453525fee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:14.264355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9879be32-ed3e-46e9-9d1e-b70453525fee admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:14.321179 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e20ad876-aebe-4d81-bc51-a669f651328b admin admin] The token cache is being invalidate because project 29ca46f1519348d9b356f7e7e0aa0947 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:14.321648 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e20ad876-aebe-4d81-bc51-a669f651328b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:14.322642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e20ad876-aebe-4d81-bc51-a669f651328b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:14.325006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 980/1969] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:14 2022] DELETE /identity/v3/projects/29ca46f1519348d9b356f7e7e0aa0947 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:14.329815 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8035b961-510f-4d89-bbd6-a5fdd423b470 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.330123 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8035b961-510f-4d89-bbd6-a5fdd423b470 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.330375 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8035b961-510f-4d89-bbd6-a5fdd423b470 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.355225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-8035b961-510f-4d89-bbd6-a5fdd423b470 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:14.356507 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8035b961-510f-4d89-bbd6-a5fdd423b470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:14.364995 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 981/1970] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:14.374004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7cd4d0-98b3-45f1-a48b-2e883c9e913d None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.374283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7cd4d0-98b3-45f1-a48b-2e883c9e913d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.374576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7cd4d0-98b3-45f1-a48b-2e883c9e913d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.395283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-1e7cd4d0-98b3-45f1-a48b-2e883c9e913d None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:14.396534 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-1e7cd4d0-98b3-45f1-a48b-2e883c9e913d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:14.405099 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 982/1971] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:14.413487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7cd4d0-98b3-45f1-a48b-2e883c9e913d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:14.421172 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 990/1972] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:14 2022] POST /identity/v3/users => generated 293 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:14.426005 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9879be32-ed3e-46e9-9d1e-b70453525fee admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:14.427632 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7cd4d0-98b3-45f1-a48b-2e883c9e913d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:14.427749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9879be32-ed3e-46e9-9d1e-b70453525fee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:14.432430 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a26560-c8cd-49d6-a16b-1529d71283de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:14.433039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a26560-c8cd-49d6-a16b-1529d71283de admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.433311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a26560-c8cd-49d6-a16b-1529d71283de admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.433578 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a26560-c8cd-49d6-a16b-1529d71283de admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.434215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a26560-c8cd-49d6-a16b-1529d71283de admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:26:14.434508 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a26560-c8cd-49d6-a16b-1529d71283de admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:14.436088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a26560-c8cd-49d6-a16b-1529d71283de admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:14.440412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9879be32-ed3e-46e9-9d1e-b70453525fee admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:14.445744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7b3b9e-8a1d-4371-b8c4-4852659be093 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:14.446094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7b3b9e-8a1d-4371-b8c4-4852659be093 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.448799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7b3b9e-8a1d-4371-b8c4-4852659be093 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.448799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7b3b9e-8a1d-4371-b8c4-4852659be093 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.448799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7b3b9e-8a1d-4371-b8c4-4852659be093 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:14.449042 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 983/1973] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:26:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:26:14.449118 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7b3b9e-8a1d-4371-b8c4-4852659be093 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:14.452857 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a26560-c8cd-49d6-a16b-1529d71283de admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:14.462878 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 991/1974] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:14 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:14.464200 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7b3b9e-8a1d-4371-b8c4-4852659be093 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:14.465740 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a26560-c8cd-49d6-a16b-1529d71283de admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:14.470960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6243f001-732c-47b5-a63f-d74a8e5029d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:14.472162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6243f001-732c-47b5-a63f-d74a8e5029d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.479351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6243f001-732c-47b5-a63f-d74a8e5029d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.479351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6243f001-732c-47b5-a63f-d74a8e5029d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.479351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6243f001-732c-47b5-a63f-d74a8e5029d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:14.479608 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7b3b9e-8a1d-4371-b8c4-4852659be093 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:14.479715 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6243f001-732c-47b5-a63f-d74a8e5029d2 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:14.479792 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff06230-cabb-4630-973b-80c3fe17ff44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:14.480231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff06230-cabb-4630-973b-80c3fe17ff44 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.480618 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff06230-cabb-4630-973b-80c3fe17ff44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.480919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff06230-cabb-4630-973b-80c3fe17ff44 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.481601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff06230-cabb-4630-973b-80c3fe17ff44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:14.483304 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff06230-cabb-4630-973b-80c3fe17ff44 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:14.499071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 992/1975] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:14 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:14.500466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff06230-cabb-4630-973b-80c3fe17ff44 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:14.512145 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff06230-cabb-4630-973b-80c3fe17ff44 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:14.516949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3324ac47-4cf7-4412-ac95-f3f76da6bb63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:14.517606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3324ac47-4cf7-4412-ac95-f3f76da6bb63 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.517955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3324ac47-4cf7-4412-ac95-f3f76da6bb63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.518180 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3324ac47-4cf7-4412-ac95-f3f76da6bb63 admin admin] PATH_INFO: `/v3/projects/7f546f58e4414ad79d1f2738f4a6ebe0/users/9c220f2e2ec344c4a0f7e1889aa1646c/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.534367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3324ac47-4cf7-4412-ac95-f3f76da6bb63 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f546f58e4414ad79d1f2738f4a6ebe0, user_id=9c220f2e2ec344c4a0f7e1889aa1646c, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:14.537515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3324ac47-4cf7-4412-ac95-f3f76da6bb63 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:14.572242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 984/1976] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:14 2022] POST /identity/v3/projects => generated 373 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:14.576719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6243f001-732c-47b5-a63f-d74a8e5029d2 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:14.633660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 993/1977] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:14 2022] PUT /identity/v3/projects/7f546f58e4414ad79d1f2738f4a6ebe0/users/9c220f2e2ec344c4a0f7e1889aa1646c/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:14.641194 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6243f001-732c-47b5-a63f-d74a8e5029d2 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:14.646006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04492e7a-6fc4-4326-996c-2b58bd15798b None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.646310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04492e7a-6fc4-4326-996c-2b58bd15798b None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.646563 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04492e7a-6fc4-4326-996c-2b58bd15798b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.683212 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-04492e7a-6fc4-4326-996c-2b58bd15798b None None] MFA Rules not processed for user `9c220f2e2ec344c4a0f7e1889aa1646c`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:14.684443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-04492e7a-6fc4-4326-996c-2b58bd15798b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:14.714034 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab8b74b-e409-4f89-a40e-5a26e8695ebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:14.714657 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab8b74b-e409-4f89-a40e-5a26e8695ebe admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.714930 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab8b74b-e409-4f89-a40e-5a26e8695ebe admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.715195 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab8b74b-e409-4f89-a40e-5a26e8695ebe admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab8b74b-e409-4f89-a40e-5a26e8695ebe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:14.717906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab8b74b-e409-4f89-a40e-5a26e8695ebe admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:14.730540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:INSERT=2,keystone:DELETE=11,keystone:SELECT=87 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:14.767446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:DELETE=10,keystone:SELECT=87,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:14.795452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 994/1978] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:14 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:14.801069 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04492e7a-6fc4-4326-996c-2b58bd15798b None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:14.812016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04492e7a-6fc4-4326-996c-2b58bd15798b None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:14.816954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee2f46e-79a1-4585-8d17-f04710421be1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:14.818294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee2f46e-79a1-4585-8d17-f04710421be1 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.818294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee2f46e-79a1-4585-8d17-f04710421be1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.818294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee2f46e-79a1-4585-8d17-f04710421be1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.821939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee2f46e-79a1-4585-8d17-f04710421be1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:14.822023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 985/1979] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:14 2022] POST /identity/v3/users => generated 287 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:14.822125 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee2f46e-79a1-4585-8d17-f04710421be1 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:14.825101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab8b74b-e409-4f89-a40e-5a26e8695ebe admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:14.843030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab8b74b-e409-4f89-a40e-5a26e8695ebe admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:14.850932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc97142-880d-41d1-9548-7c20b2f911a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:14.851690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc97142-880d-41d1-9548-7c20b2f911a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.852061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc97142-880d-41d1-9548-7c20b2f911a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.852504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc97142-880d-41d1-9548-7c20b2f911a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.853293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc97142-880d-41d1-9548-7c20b2f911a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:14.855349 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc97142-880d-41d1-9548-7c20b2f911a8 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:14.870207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 986/1980] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:14 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:14.876013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc97142-880d-41d1-9548-7c20b2f911a8 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:14.889919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc97142-880d-41d1-9548-7c20b2f911a8 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:14.896463 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177fea97-2b57-462a-bb25-7622c181d3d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:14.897093 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177fea97-2b57-462a-bb25-7622c181d3d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.897372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177fea97-2b57-462a-bb25-7622c181d3d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.897647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177fea97-2b57-462a-bb25-7622c181d3d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.898353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177fea97-2b57-462a-bb25-7622c181d3d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:14.900162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177fea97-2b57-462a-bb25-7622c181d3d0 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:14.916512 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 987/1981] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:14 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:14.921302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177fea97-2b57-462a-bb25-7622c181d3d0 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:14.932161 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177fea97-2b57-462a-bb25-7622c181d3d0 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:14.937022 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b33b51-0172-4d96-8e1b-81de7dc86b48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:14.937678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b33b51-0172-4d96-8e1b-81de7dc86b48 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:14.937962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b33b51-0172-4d96-8e1b-81de7dc86b48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:14.938232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b33b51-0172-4d96-8e1b-81de7dc86b48 admin admin] PATH_INFO: `/v3/projects/7525c268e881425aaf3b6fa530b85ab3/users/b1f05b8f41b440009c7a2ad6b698b758/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:14.954483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b33b51-0172-4d96-8e1b-81de7dc86b48 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7525c268e881425aaf3b6fa530b85ab3, user_id=b1f05b8f41b440009c7a2ad6b698b758, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:14.957992 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b33b51-0172-4d96-8e1b-81de7dc86b48 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:14.992331 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 995/1982] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:14 2022] POST /identity/v3/projects => generated 383 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:14.995999 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee2f46e-79a1-4585-8d17-f04710421be1 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:15.060991 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee2f46e-79a1-4585-8d17-f04710421be1 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:15.064989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 988/1983] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:14 2022] PUT /identity/v3/projects/7525c268e881425aaf3b6fa530b85ab3/users/b1f05b8f41b440009c7a2ad6b698b758/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:15.073962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262bd7d5-e127-4da6-a49f-a1dd57ac84e8 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:15.074212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262bd7d5-e127-4da6-a49f-a1dd57ac84e8 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:15.074595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262bd7d5-e127-4da6-a49f-a1dd57ac84e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:15.094340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-262bd7d5-e127-4da6-a49f-a1dd57ac84e8 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:15.095562 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-262bd7d5-e127-4da6-a49f-a1dd57ac84e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:15.120319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17736289-0767-4c43-a7a9-d040c734c0ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:15.120943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17736289-0767-4c43-a7a9-d040c734c0ab admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:15.135645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17736289-0767-4c43-a7a9-d040c734c0ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:15.135645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17736289-0767-4c43-a7a9-d040c734c0ab admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:15.135645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17736289-0767-4c43-a7a9-d040c734c0ab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:15.135645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17736289-0767-4c43-a7a9-d040c734c0ab admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:15.163927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 989/1984] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:15.173579 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262bd7d5-e127-4da6-a49f-a1dd57ac84e8 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:15.175028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-262bd7d5-e127-4da6-a49f-a1dd57ac84e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:15.237364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262bd7d5-e127-4da6-a49f-a1dd57ac84e8 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:15.241424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126cf610-d4f7-4816-a76b-5fbc963d9d50 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:15.242061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126cf610-d4f7-4816-a76b-5fbc963d9d50 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:15.242354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126cf610-d4f7-4816-a76b-5fbc963d9d50 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:15.242625 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126cf610-d4f7-4816-a76b-5fbc963d9d50 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:15.255137 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126cf610-d4f7-4816-a76b-5fbc963d9d50 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:15.257321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126cf610-d4f7-4816-a76b-5fbc963d9d50 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:15.267239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 996/1985] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:14 2022] POST /identity/v3/users => generated 298 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:15.269057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17736289-0767-4c43-a7a9-d040c734c0ab admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:15.272760 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 990/1986] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:15.279457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17736289-0767-4c43-a7a9-d040c734c0ab admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:15.284324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af69d5a-1ee6-4f53-a699-81869a5b0285 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:15.284936 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af69d5a-1ee6-4f53-a699-81869a5b0285 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:15.285216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af69d5a-1ee6-4f53-a699-81869a5b0285 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:15.285485 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af69d5a-1ee6-4f53-a699-81869a5b0285 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:15.286125 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af69d5a-1ee6-4f53-a699-81869a5b0285 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:15.287972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af69d5a-1ee6-4f53-a699-81869a5b0285 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:15.298438 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 997/1987] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:15.304155 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126cf610-d4f7-4816-a76b-5fbc963d9d50 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:15.315411 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126cf610-d4f7-4816-a76b-5fbc963d9d50 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:15.320700 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a4915b-c86d-457c-8318-1b0921360f53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:15.321480 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a4915b-c86d-457c-8318-1b0921360f53 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:15.321762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a4915b-c86d-457c-8318-1b0921360f53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:15.322058 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a4915b-c86d-457c-8318-1b0921360f53 admin admin] PATH_INFO: `/v3/projects/95426b1bdac44ce1817fed1ced87fe1c/users/04e7534ac5964861ae7ce4c6efa59c2a/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:15.339073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a4915b-c86d-457c-8318-1b0921360f53 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95426b1bdac44ce1817fed1ced87fe1c, user_id=04e7534ac5964861ae7ce4c6efa59c2a, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:15.341915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a4915b-c86d-457c-8318-1b0921360f53 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:15.404927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:15.452601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 991/1988] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:15 2022] PUT /identity/v3/projects/95426b1bdac44ce1817fed1ced87fe1c/users/04e7534ac5964861ae7ce4c6efa59c2a/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:15.460139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af69d5a-1ee6-4f53-a699-81869a5b0285 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:15.467488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:15.476023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af69d5a-1ee6-4f53-a699-81869a5b0285 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:15.534124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953bbf3a-0cef-4fac-a059-58322829d845 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:15.534640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953bbf3a-0cef-4fac-a059-58322829d845 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:15.534912 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953bbf3a-0cef-4fac-a059-58322829d845 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:15.535178 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953bbf3a-0cef-4fac-a059-58322829d845 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:15.535811 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953bbf3a-0cef-4fac-a059-58322829d845 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:15.537709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953bbf3a-0cef-4fac-a059-58322829d845 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:15.547958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 998/1989] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:15 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:15.553010 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a4915b-c86d-457c-8318-1b0921360f53 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:15.563674 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a4915b-c86d-457c-8318-1b0921360f53 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:15.569130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edf4386-3d7c-4ee7-88c7-baa366d81dad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:15.569804 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edf4386-3d7c-4ee7-88c7-baa366d81dad admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:15.570223 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edf4386-3d7c-4ee7-88c7-baa366d81dad admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:15.570492 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edf4386-3d7c-4ee7-88c7-baa366d81dad admin admin] PATH_INFO: `/v3/projects/95426b1bdac44ce1817fed1ced87fe1c/users/04e7534ac5964861ae7ce4c6efa59c2a/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:15.572616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edf4386-3d7c-4ee7-88c7-baa366d81dad admin admin] RBAC: Authorizing `identity:create_grant(project_id=95426b1bdac44ce1817fed1ced87fe1c, user_id=04e7534ac5964861ae7ce4c6efa59c2a, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:15.575543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edf4386-3d7c-4ee7-88c7-baa366d81dad admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:15.601318 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 992/1990] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:15 2022] PUT /identity/v3/projects/95426b1bdac44ce1817fed1ced87fe1c/users/04e7534ac5964861ae7ce4c6efa59c2a/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:15.613215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4030f45-58dc-4448-b38c-8cf103392354 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:15.613506 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4030f45-58dc-4448-b38c-8cf103392354 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:15.613880 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4030f45-58dc-4448-b38c-8cf103392354 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:15.639848 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:15.645205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-a4030f45-58dc-4448-b38c-8cf103392354 None None] MFA Rules not processed for user `04e7534ac5964861ae7ce4c6efa59c2a`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:15.648246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-a4030f45-58dc-4448-b38c-8cf103392354 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:15.706039 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:15.720953 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 999/1991] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:15 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:15.731979 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edf4386-3d7c-4ee7-88c7-baa366d81dad admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:15.733629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9edf4386-3d7c-4ee7-88c7-baa366d81dad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:15.801341 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edf4386-3d7c-4ee7-88c7-baa366d81dad admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:15.808673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85747f6c-d93d-4bc0-a08c-49f7204886ce service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:15.810121 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85747f6c-d93d-4bc0-a08c-49f7204886ce service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:15.810642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85747f6c-d93d-4bc0-a08c-49f7204886ce service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:15.810986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85747f6c-d93d-4bc0-a08c-49f7204886ce service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:15.858524 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:15.897382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85747f6c-d93d-4bc0-a08c-49f7204886ce service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:15.899874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85747f6c-d93d-4bc0-a08c-49f7204886ce service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:15.911176 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 993/1992] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:15 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:16.047519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:16.558863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:16.649192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a587ac70-4bfd-4d1b-b083-56a3659bd7ee None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:16.650173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a587ac70-4bfd-4d1b-b083-56a3659bd7ee None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:16.650616 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a587ac70-4bfd-4d1b-b083-56a3659bd7ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:16.692076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-a587ac70-4bfd-4d1b-b083-56a3659bd7ee None None] MFA Rules not processed for user `b1f05b8f41b440009c7a2ad6b698b758`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:16.693810 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-a587ac70-4bfd-4d1b-b083-56a3659bd7ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:16.781619 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1000/1993] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:16 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:16.786970 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85747f6c-d93d-4bc0-a08c-49f7204886ce service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:16.862709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85747f6c-d93d-4bc0-a08c-49f7204886ce service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:16.867523 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3405b501-3db4-44bc-ac4f-567022703b00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:16.868227 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3405b501-3db4-44bc-ac4f-567022703b00 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:16.868505 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3405b501-3db4-44bc-ac4f-567022703b00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:16.868879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3405b501-3db4-44bc-ac4f-567022703b00 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:16.869387 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3405b501-3db4-44bc-ac4f-567022703b00 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:16.871466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3405b501-3db4-44bc-ac4f-567022703b00 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:17.084839 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 994/1994] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:16 2022] POST /identity/v3/projects => generated 371 bytes in 299 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:17.089142 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a587ac70-4bfd-4d1b-b083-56a3659bd7ee None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:17.101089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a587ac70-4bfd-4d1b-b083-56a3659bd7ee None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:17.161771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b797dbb-4e3d-4944-abf8-84c1ec7ce8e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:17.162513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b797dbb-4e3d-4944-abf8-84c1ec7ce8e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:17.162794 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b797dbb-4e3d-4944-abf8-84c1ec7ce8e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:17.163141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b797dbb-4e3d-4944-abf8-84c1ec7ce8e0 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:17.163788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b797dbb-4e3d-4944-abf8-84c1ec7ce8e0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:17.165829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b797dbb-4e3d-4944-abf8-84c1ec7ce8e0 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:17.261938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1001/1995] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:17 2022] POST /identity/v3/users => generated 292 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:17.265991 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3405b501-3db4-44bc-ac4f-567022703b00 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:17.278233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3405b501-3db4-44bc-ac4f-567022703b00 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:17.283048 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad161fd1-f9fb-4d68-89d8-8b02bee8538c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:17.283653 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad161fd1-f9fb-4d68-89d8-8b02bee8538c admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:17.283931 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad161fd1-f9fb-4d68-89d8-8b02bee8538c admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:17.284218 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad161fd1-f9fb-4d68-89d8-8b02bee8538c admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:17.285024 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad161fd1-f9fb-4d68-89d8-8b02bee8538c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:17.287908 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad161fd1-f9fb-4d68-89d8-8b02bee8538c admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:17.299562 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 995/1996] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:17 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:17.305032 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b797dbb-4e3d-4944-abf8-84c1ec7ce8e0 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:17.316933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b797dbb-4e3d-4944-abf8-84c1ec7ce8e0 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:17.322019 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0171cbb-7d62-4da4-bef2-686da20d7c32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:17.322743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0171cbb-7d62-4da4-bef2-686da20d7c32 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:17.323097 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0171cbb-7d62-4da4-bef2-686da20d7c32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:17.323441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0171cbb-7d62-4da4-bef2-686da20d7c32 admin admin] PATH_INFO: `/v3/projects/c1299e8a288741779f68680d3e3d07cd/users/6462955967a84063b74d7f5b8770601b/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:17.340244 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0171cbb-7d62-4da4-bef2-686da20d7c32 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1299e8a288741779f68680d3e3d07cd, user_id=6462955967a84063b74d7f5b8770601b, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:17.343422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0171cbb-7d62-4da4-bef2-686da20d7c32 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:17.389923 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:17.441919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1002/1997] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:17 2022] PUT /identity/v3/projects/c1299e8a288741779f68680d3e3d07cd/users/6462955967a84063b74d7f5b8770601b/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:17.446104 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad161fd1-f9fb-4d68-89d8-8b02bee8538c admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:17.457677 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad161fd1-f9fb-4d68-89d8-8b02bee8538c admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:17.517692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5cc2c99-1640-40ef-a792-a0383cee7496 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:17.518307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cc2c99-1640-40ef-a792-a0383cee7496 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:17.518588 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cc2c99-1640-40ef-a792-a0383cee7496 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:17.518862 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cc2c99-1640-40ef-a792-a0383cee7496 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:17.519501 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5cc2c99-1640-40ef-a792-a0383cee7496 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:17.521844 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5cc2c99-1640-40ef-a792-a0383cee7496 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:17.532493 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 996/1998] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:17 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:17.537150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0171cbb-7d62-4da4-bef2-686da20d7c32 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:17.547446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0171cbb-7d62-4da4-bef2-686da20d7c32 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:17.552289 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56d1c37-3c5b-4dd4-aa1f-e37493a7a302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:17.552942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56d1c37-3c5b-4dd4-aa1f-e37493a7a302 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:17.553218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56d1c37-3c5b-4dd4-aa1f-e37493a7a302 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:17.553491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56d1c37-3c5b-4dd4-aa1f-e37493a7a302 admin admin] PATH_INFO: `/v3/projects/c1299e8a288741779f68680d3e3d07cd/users/6462955967a84063b74d7f5b8770601b/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:17.555417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56d1c37-3c5b-4dd4-aa1f-e37493a7a302 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1299e8a288741779f68680d3e3d07cd, user_id=6462955967a84063b74d7f5b8770601b, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:17.558208 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56d1c37-3c5b-4dd4-aa1f-e37493a7a302 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:17.570131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1003/1999] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:17 2022] PUT /identity/v3/projects/c1299e8a288741779f68680d3e3d07cd/users/6462955967a84063b74d7f5b8770601b/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:18.345332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=87,keystone:DELETE=10,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:18.435587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:19.174096 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c345203-91b3-4fe5-9e72-8755fd205633 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:19.174377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c345203-91b3-4fe5-9e72-8755fd205633 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:19.174831 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c345203-91b3-4fe5-9e72-8755fd205633 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:19.204952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-6c345203-91b3-4fe5-9e72-8755fd205633 None None] MFA Rules not processed for user `6462955967a84063b74d7f5b8770601b`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:19.206175 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-6c345203-91b3-4fe5-9e72-8755fd205633 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:19.287697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 997/2000] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:19 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:19.299756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56d1c37-3c5b-4dd4-aa1f-e37493a7a302 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:19.301613 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e56d1c37-3c5b-4dd4-aa1f-e37493a7a302 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:19.362263 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56d1c37-3c5b-4dd4-aa1f-e37493a7a302 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:19.366045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da1905b-2fcc-47a8-88ad-106135740d9c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:19.366655 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da1905b-2fcc-47a8-88ad-106135740d9c service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:19.367056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da1905b-2fcc-47a8-88ad-106135740d9c service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:19.367195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da1905b-2fcc-47a8-88ad-106135740d9c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:19.426568 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da1905b-2fcc-47a8-88ad-106135740d9c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:19.428015 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da1905b-2fcc-47a8-88ad-106135740d9c service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:19.437268 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1004/2001] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:19 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:19.668971 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:19.851247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:21.161566 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:21.421118 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c345203-91b3-4fe5-9e72-8755fd205633 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:21.429450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:21.433240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c345203-91b3-4fe5-9e72-8755fd205633 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:21.437263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18b42e7-3bda-41e9-b891-6ee4919d553f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:21.437901 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b42e7-3bda-41e9-b891-6ee4919d553f service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:21.438176 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b42e7-3bda-41e9-b891-6ee4919d553f service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:21.438445 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b42e7-3bda-41e9-b891-6ee4919d553f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:21.448737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18b42e7-3bda-41e9-b891-6ee4919d553f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:21.450257 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18b42e7-3bda-41e9-b891-6ee4919d553f service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:21.461178 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 998/2002] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:21 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:22.636664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da1905b-2fcc-47a8-88ad-106135740d9c service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:22.647638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da1905b-2fcc-47a8-88ad-106135740d9c service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:22.651711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34ee5dd-1df2-403f-ad12-576442ae6527 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:22.652366 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34ee5dd-1df2-403f-ad12-576442ae6527 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:22.652641 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34ee5dd-1df2-403f-ad12-576442ae6527 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:22.652908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34ee5dd-1df2-403f-ad12-576442ae6527 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:22.657033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:22.719966 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34ee5dd-1df2-403f-ad12-576442ae6527 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:22.721467 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34ee5dd-1df2-403f-ad12-576442ae6527 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:22.734190 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1005/2003] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:22 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:22.810814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:23.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:23.789816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:24.109190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:24.326214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:24.449706 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:24.515353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:24.786470 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:INSERT=3,keystone:DELETE=8 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:24.907253 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:25.331849 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:25.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:25.509185 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:25.561803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:25.906546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:29.412905 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18b42e7-3bda-41e9-b891-6ee4919d553f service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:29.436588 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:29.500674 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18b42e7-3bda-41e9-b891-6ee4919d553f service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:29.505887 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c210a145-8825-4202-92ba-0fcb908f86f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:29.506694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c210a145-8825-4202-92ba-0fcb908f86f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:29.507076 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c210a145-8825-4202-92ba-0fcb908f86f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:29.507604 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c210a145-8825-4202-92ba-0fcb908f86f3 admin admin] PATH_INFO: `/v3/users/9c220f2e2ec344c4a0f7e1889aa1646c` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:29.509185 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c210a145-8825-4202-92ba-0fcb908f86f3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9c220f2e2ec344c4a0f7e1889aa1646c)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:29.511322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c210a145-8825-4202-92ba-0fcb908f86f3 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:29.784277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c210a145-8825-4202-92ba-0fcb908f86f3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9c220f2e2ec344c4a0f7e1889aa1646c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:29.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c210a145-8825-4202-92ba-0fcb908f86f3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9c220f2e2ec344c4a0f7e1889aa1646c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:29.794233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c210a145-8825-4202-92ba-0fcb908f86f3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9c220f2e2ec344c4a0f7e1889aa1646c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:29.843170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c210a145-8825-4202-92ba-0fcb908f86f3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9c220f2e2ec344c4a0f7e1889aa1646c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:29.965953 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c210a145-8825-4202-92ba-0fcb908f86f3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9c220f2e2ec344c4a0f7e1889aa1646c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:29.976876 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 999/2004] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:29 2022] DELETE /identity/v3/users/9c220f2e2ec344c4a0f7e1889aa1646c => generated 0 bytes in 565 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:29.984144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7a7649-bb19-40f4-924b-674f54c1b458 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:29.986167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7a7649-bb19-40f4-924b-674f54c1b458 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:29.987957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7a7649-bb19-40f4-924b-674f54c1b458 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:30.024043 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-1e7a7649-bb19-40f4-924b-674f54c1b458 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:30.026508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1e7a7649-bb19-40f4-924b-674f54c1b458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:30.132351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1006/2005] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:30.149807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c210a145-8825-4202-92ba-0fcb908f86f3 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:30.151984 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-c210a145-8825-4202-92ba-0fcb908f86f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:30.260088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c210a145-8825-4202-92ba-0fcb908f86f3 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:30.269502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86b8733-df3d-48af-bf4c-4e74b6e89c5a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:30.269502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86b8733-df3d-48af-bf4c-4e74b6e89c5a service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:30.269502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86b8733-df3d-48af-bf4c-4e74b6e89c5a service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:30.269502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86b8733-df3d-48af-bf4c-4e74b6e89c5a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:30.278759 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86b8733-df3d-48af-bf4c-4e74b6e89c5a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:30.280402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86b8733-df3d-48af-bf4c-4e74b6e89c5a service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:30.293261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1000/2006] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:30.844539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7a7649-bb19-40f4-924b-674f54c1b458 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:30.846437 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1e7a7649-bb19-40f4-924b-674f54c1b458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:30.874001 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7a7649-bb19-40f4-924b-674f54c1b458 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:30.879946 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5d1106-6280-49fc-84ca-9000084f65fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:30.880818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5d1106-6280-49fc-84ca-9000084f65fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:30.881240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5d1106-6280-49fc-84ca-9000084f65fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:30.881510 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5d1106-6280-49fc-84ca-9000084f65fb admin admin] PATH_INFO: `/v3/projects/7f546f58e4414ad79d1f2738f4a6ebe0` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:30.882906 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5d1106-6280-49fc-84ca-9000084f65fb admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f546f58e4414ad79d1f2738f4a6ebe0)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:30.885007 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5d1106-6280-49fc-84ca-9000084f65fb admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:30.944053 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f6ab9c-464e-4d6a-88fa-b525cdf79632 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:30.944743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f6ab9c-464e-4d6a-88fa-b525cdf79632 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:30.944883 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f6ab9c-464e-4d6a-88fa-b525cdf79632 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:30.946280 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1001/2007] 10.222.0.47 () {54 vars in 921 bytes} [Sun Aug 21 01:26:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:26:30.954197 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e229a2-528a-4a71-bcce-3c3b5789e330 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:30.954759 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e229a2-528a-4a71-bcce-3c3b5789e330 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:30.955523 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e229a2-528a-4a71-bcce-3c3b5789e330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:30.983478 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-95e229a2-528a-4a71-bcce-3c3b5789e330 None None] MFA Rules not processed for user `972b0f1dbfef4d85a6dfc4c5fd839bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:30.985066 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-95e229a2-528a-4a71-bcce-3c3b5789e330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:30.996157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0f5d1106-6280-49fc-84ca-9000084f65fb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f546f58e4414ad79d1f2738f4a6ebe0', 'request_id': 'req-0f5d1106-6280-49fc-84ca-9000084f65fb'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:31.029272 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:31.065629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1002/2008] 10.222.0.47 () {58 vars in 1018 bytes} [Sun Aug 21 01:26:30 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:31.075616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e229a2-528a-4a71-bcce-3c3b5789e330 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:31.077105 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-95e229a2-528a-4a71-bcce-3c3b5789e330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:31.090281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e229a2-528a-4a71-bcce-3c3b5789e330 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:31.095552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ce64bf-179c-4059-ab07-f1e87c4c6ba5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '972b0f1dbfef4d85a6dfc4c5fd839bde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:31.096189 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ce64bf-179c-4059-ab07-f1e87c4c6ba5 service placement] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:31.096504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ce64bf-179c-4059-ab07-f1e87c4c6ba5 service placement] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:31.096769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ce64bf-179c-4059-ab07-f1e87c4c6ba5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:31.105133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ce64bf-179c-4059-ab07-f1e87c4c6ba5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:31.106754 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ce64bf-179c-4059-ab07-f1e87c4c6ba5 service placement] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:31.113815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0f5d1106-6280-49fc-84ca-9000084f65fb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f546f58e4414ad79d1f2738f4a6ebe0', 'request_id': 'req-0f5d1106-6280-49fc-84ca-9000084f65fb'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:31.117212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1003/2009] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:31 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:26:31.122907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0f5d1106-6280-49fc-84ca-9000084f65fb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f546f58e4414ad79d1f2738f4a6ebe0', 'request_id': 'req-0f5d1106-6280-49fc-84ca-9000084f65fb'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:31.127495 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ce64bf-179c-4059-ab07-f1e87c4c6ba5 service placement] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:31.131974 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-25ce64bf-179c-4059-ab07-f1e87c4c6ba5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:31.138460 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0f5d1106-6280-49fc-84ca-9000084f65fb admin admin] The token cache is being invalidate because project 7f546f58e4414ad79d1f2738f4a6ebe0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:31.138938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0f5d1106-6280-49fc-84ca-9000084f65fb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:31.139750 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0f5d1106-6280-49fc-84ca-9000084f65fb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:31.142967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1007/2010] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:30 2022] DELETE /identity/v3/projects/7f546f58e4414ad79d1f2738f4a6ebe0 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:31.147659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5d1106-6280-49fc-84ca-9000084f65fb admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:31.151334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-0f5d1106-6280-49fc-84ca-9000084f65fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:31.162672 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:31.211051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ce64bf-179c-4059-ab07-f1e87c4c6ba5 service placement] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:31.216310 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2293e0e0-547a-41d1-a980-eeab36fe82d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:31.216970 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2293e0e0-547a-41d1-a980-eeab36fe82d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:31.217209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2293e0e0-547a-41d1-a980-eeab36fe82d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:31.217475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2293e0e0-547a-41d1-a980-eeab36fe82d7 admin admin] PATH_INFO: `/v3/users/bc7137d0ba634b6ab4b83de1ab47c476` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:31.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2293e0e0-547a-41d1-a980-eeab36fe82d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc7137d0ba634b6ab4b83de1ab47c476)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:31.219940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5d1106-6280-49fc-84ca-9000084f65fb admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:31.220561 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2293e0e0-547a-41d1-a980-eeab36fe82d7 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:31.226641 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfddb301-e3a7-4c57-9367-fe875d01e15b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:31.227345 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfddb301-e3a7-4c57-9367-fe875d01e15b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:31.227627 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfddb301-e3a7-4c57-9367-fe875d01e15b admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:31.228019 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfddb301-e3a7-4c57-9367-fe875d01e15b admin admin] PATH_INFO: `/v3/users/04e7534ac5964861ae7ce4c6efa59c2a` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:31.229430 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfddb301-e3a7-4c57-9367-fe875d01e15b admin admin] RBAC: Authorizing `identity:delete_user(user_id=04e7534ac5964861ae7ce4c6efa59c2a)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:31.231565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfddb301-e3a7-4c57-9367-fe875d01e15b admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:31.442857 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-2293e0e0-547a-41d1-a980-eeab36fe82d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc7137d0ba634b6ab4b83de1ab47c476'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:31.449893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-2293e0e0-547a-41d1-a980-eeab36fe82d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc7137d0ba634b6ab4b83de1ab47c476'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:31.454577 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-2293e0e0-547a-41d1-a980-eeab36fe82d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc7137d0ba634b6ab4b83de1ab47c476'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:31.463424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-2293e0e0-547a-41d1-a980-eeab36fe82d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc7137d0ba634b6ab4b83de1ab47c476'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:31.472699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-cfddb301-e3a7-4c57-9367-fe875d01e15b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '04e7534ac5964861ae7ce4c6efa59c2a'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:31.481428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-cfddb301-e3a7-4c57-9367-fe875d01e15b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '04e7534ac5964861ae7ce4c6efa59c2a'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:31.494710 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-cfddb301-e3a7-4c57-9367-fe875d01e15b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '04e7534ac5964861ae7ce4c6efa59c2a'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:31.554610 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-2293e0e0-547a-41d1-a980-eeab36fe82d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc7137d0ba634b6ab4b83de1ab47c476'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:31.561183 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1004/2011] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:31 2022] DELETE /identity/v3/users/bc7137d0ba634b6ab4b83de1ab47c476 => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:31.654601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-cfddb301-e3a7-4c57-9367-fe875d01e15b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '04e7534ac5964861ae7ce4c6efa59c2a'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:31.655249 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-cfddb301-e3a7-4c57-9367-fe875d01e15b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '04e7534ac5964861ae7ce4c6efa59c2a'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:31.663263 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1008/2012] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:31 2022] DELETE /identity/v3/users/04e7534ac5964861ae7ce4c6efa59c2a => generated 0 bytes in 516 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:31.816361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2293e0e0-547a-41d1-a980-eeab36fe82d7 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:31.817819 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-2293e0e0-547a-41d1-a980-eeab36fe82d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:31.896365 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:31.940999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2293e0e0-547a-41d1-a980-eeab36fe82d7 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:32.018042 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58eb38f5-5e6e-47e2-ba3c-39808021f746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:32.019482 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58eb38f5-5e6e-47e2-ba3c-39808021f746 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:32.020424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58eb38f5-5e6e-47e2-ba3c-39808021f746 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:32.020866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58eb38f5-5e6e-47e2-ba3c-39808021f746 admin admin] PATH_INFO: `/v3/projects/a1914f8232864af6ad97aebcb1636125` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:32.022437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58eb38f5-5e6e-47e2-ba3c-39808021f746 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a1914f8232864af6ad97aebcb1636125)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:32.024908 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58eb38f5-5e6e-47e2-ba3c-39808021f746 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:32.163696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-58eb38f5-5e6e-47e2-ba3c-39808021f746 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a1914f8232864af6ad97aebcb1636125', 'request_id': 'req-58eb38f5-5e6e-47e2-ba3c-39808021f746'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:32.163998 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfddb301-e3a7-4c57-9367-fe875d01e15b admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:32.165525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-cfddb301-e3a7-4c57-9367-fe875d01e15b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:32.170926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-58eb38f5-5e6e-47e2-ba3c-39808021f746 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a1914f8232864af6ad97aebcb1636125', 'request_id': 'req-58eb38f5-5e6e-47e2-ba3c-39808021f746'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:32.178940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-58eb38f5-5e6e-47e2-ba3c-39808021f746 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a1914f8232864af6ad97aebcb1636125', 'request_id': 'req-58eb38f5-5e6e-47e2-ba3c-39808021f746'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:32.187162 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfddb301-e3a7-4c57-9367-fe875d01e15b admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:32.256518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-58eb38f5-5e6e-47e2-ba3c-39808021f746 admin admin] The token cache is being invalidate because project a1914f8232864af6ad97aebcb1636125 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:32.256899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-58eb38f5-5e6e-47e2-ba3c-39808021f746 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:32.257774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-58eb38f5-5e6e-47e2-ba3c-39808021f746 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:32.259281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1005/2013] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:31 2022] DELETE /identity/v3/projects/a1914f8232864af6ad97aebcb1636125 => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:32.260195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb2ec70-6bdb-40a3-a475-6e74c0439ac1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:32.263950 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb2ec70-6bdb-40a3-a475-6e74c0439ac1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:32.263950 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb2ec70-6bdb-40a3-a475-6e74c0439ac1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:32.263950 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb2ec70-6bdb-40a3-a475-6e74c0439ac1 admin admin] PATH_INFO: `/v3/projects/95426b1bdac44ce1817fed1ced87fe1c` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:32.263950 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb2ec70-6bdb-40a3-a475-6e74c0439ac1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95426b1bdac44ce1817fed1ced87fe1c)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:32.264633 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb2ec70-6bdb-40a3-a475-6e74c0439ac1 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:32.267438 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3af422-5dd3-4056-b457-3940ac00d0cd None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:32.267728 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3af422-5dd3-4056-b457-3940ac00d0cd None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:32.268003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3af422-5dd3-4056-b457-3940ac00d0cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:32.291825 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-8b3af422-5dd3-4056-b457-3940ac00d0cd None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:32.293133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8b3af422-5dd3-4056-b457-3940ac00d0cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:32.301666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1006/2014] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:32.310593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c336fc-e972-4b4a-9d77-0756a1c7db92 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:32.310870 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c336fc-e972-4b4a-9d77-0756a1c7db92 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:32.311137 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c336fc-e972-4b4a-9d77-0756a1c7db92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:32.331158 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-41c336fc-e972-4b4a-9d77-0756a1c7db92 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:32.332413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-41c336fc-e972-4b4a-9d77-0756a1c7db92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:32.341108 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1007/2015] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:32.347866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c336fc-e972-4b4a-9d77-0756a1c7db92 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:32.357378 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c336fc-e972-4b4a-9d77-0756a1c7db92 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:32.362288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c44eb84-9c34-4268-8034-3b8fd7899eda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:32.362894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c44eb84-9c34-4268-8034-3b8fd7899eda admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:32.363171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c44eb84-9c34-4268-8034-3b8fd7899eda admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:32.363438 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c44eb84-9c34-4268-8034-3b8fd7899eda admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:32.364077 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c44eb84-9c34-4268-8034-3b8fd7899eda admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:26:32.364439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c44eb84-9c34-4268-8034-3b8fd7899eda admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:32.366025 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c44eb84-9c34-4268-8034-3b8fd7899eda admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:32.371994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:32.378783 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1008/2016] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:26:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:26:32.383240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c44eb84-9c34-4268-8034-3b8fd7899eda admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:32.389062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-fbb2ec70-6bdb-40a3-a475-6e74c0439ac1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95426b1bdac44ce1817fed1ced87fe1c', 'request_id': 'req-fbb2ec70-6bdb-40a3-a475-6e74c0439ac1'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:32.394012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c44eb84-9c34-4268-8034-3b8fd7899eda admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:32.442557 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:32.451365 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524f9406-6f29-4d6c-a3ce-ad5681e60df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:32.452019 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524f9406-6f29-4d6c-a3ce-ad5681e60df7 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:32.452288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524f9406-6f29-4d6c-a3ce-ad5681e60df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:32.452564 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524f9406-6f29-4d6c-a3ce-ad5681e60df7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:32.459669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524f9406-6f29-4d6c-a3ce-ad5681e60df7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:32.459669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524f9406-6f29-4d6c-a3ce-ad5681e60df7 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:32.459967 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-fbb2ec70-6bdb-40a3-a475-6e74c0439ac1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95426b1bdac44ce1817fed1ced87fe1c', 'request_id': 'req-fbb2ec70-6bdb-40a3-a475-6e74c0439ac1'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:32.465418 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-fbb2ec70-6bdb-40a3-a475-6e74c0439ac1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95426b1bdac44ce1817fed1ced87fe1c', 'request_id': 'req-fbb2ec70-6bdb-40a3-a475-6e74c0439ac1'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:32.478953 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-fbb2ec70-6bdb-40a3-a475-6e74c0439ac1 admin admin] The token cache is being invalidate because project 95426b1bdac44ce1817fed1ced87fe1c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:32.479268 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-fbb2ec70-6bdb-40a3-a475-6e74c0439ac1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:32.479918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-fbb2ec70-6bdb-40a3-a475-6e74c0439ac1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:32.481441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1009/2017] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:32 2022] DELETE /identity/v3/projects/95426b1bdac44ce1817fed1ced87fe1c => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:32.487328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ff13ae-c661-40cb-9bb9-ec6fbe48b7d4 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:32.487605 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ff13ae-c661-40cb-9bb9-ec6fbe48b7d4 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:32.487873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ff13ae-c661-40cb-9bb9-ec6fbe48b7d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:32.509541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-03ff13ae-c661-40cb-9bb9-ec6fbe48b7d4 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:32.510765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-03ff13ae-c661-40cb-9bb9-ec6fbe48b7d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:32.519278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1010/2018] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:32.527521 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dc3cc6-7eb4-4258-9043-0d2a00fcab60 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:32.527800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dc3cc6-7eb4-4258-9043-0d2a00fcab60 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:32.528071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dc3cc6-7eb4-4258-9043-0d2a00fcab60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:32.547427 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-c5dc3cc6-7eb4-4258-9043-0d2a00fcab60 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:32.548704 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c5dc3cc6-7eb4-4258-9043-0d2a00fcab60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:32.556868 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1011/2019] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:32.561059 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5dc3cc6-7eb4-4258-9043-0d2a00fcab60 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:32.570515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5dc3cc6-7eb4-4258-9043-0d2a00fcab60 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:32.574728 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1009/2020] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:32 2022] POST /identity/v3/projects => generated 405 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:32.575768 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524f9406-6f29-4d6c-a3ce-ad5681e60df7 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:32.577437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-524f9406-6f29-4d6c-a3ce-ad5681e60df7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:32.628608 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e9020c-a108-43da-82fe-b0117324a036 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:32.629222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e9020c-a108-43da-82fe-b0117324a036 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:32.629498 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e9020c-a108-43da-82fe-b0117324a036 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:32.629768 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e9020c-a108-43da-82fe-b0117324a036 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:32.630443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e9020c-a108-43da-82fe-b0117324a036 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:26:32.630742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e9020c-a108-43da-82fe-b0117324a036 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:32.632463 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e9020c-a108-43da-82fe-b0117324a036 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:32.644016 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524f9406-6f29-4d6c-a3ce-ad5681e60df7 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:32.644993 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1012/2021] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:26:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:26:32.649434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e9020c-a108-43da-82fe-b0117324a036 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:32.652475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af62afe-6863-4bdf-aa39-1eaac53be218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:32.653094 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af62afe-6863-4bdf-aa39-1eaac53be218 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:32.654060 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af62afe-6863-4bdf-aa39-1eaac53be218 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:32.654060 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af62afe-6863-4bdf-aa39-1eaac53be218 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:32.654628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af62afe-6863-4bdf-aa39-1eaac53be218 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:32.656949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af62afe-6863-4bdf-aa39-1eaac53be218 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:32.659825 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e9020c-a108-43da-82fe-b0117324a036 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:32.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffee8faa-35be-42e9-8819-6a566be71e90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:32.665436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffee8faa-35be-42e9-8819-6a566be71e90 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:32.665605 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffee8faa-35be-42e9-8819-6a566be71e90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:32.665845 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffee8faa-35be-42e9-8819-6a566be71e90 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:32.666604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffee8faa-35be-42e9-8819-6a566be71e90 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:32.668672 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffee8faa-35be-42e9-8819-6a566be71e90 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:32.738116 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1010/2022] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:32 2022] POST /identity/v3/users => generated 303 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:32.742408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af62afe-6863-4bdf-aa39-1eaac53be218 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:32.752082 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af62afe-6863-4bdf-aa39-1eaac53be218 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:32.757655 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf57b00-7d07-4114-a53f-e90605ffa46e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:32.758359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf57b00-7d07-4114-a53f-e90605ffa46e admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:32.758633 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf57b00-7d07-4114-a53f-e90605ffa46e admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:32.758898 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf57b00-7d07-4114-a53f-e90605ffa46e admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:32.759533 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf57b00-7d07-4114-a53f-e90605ffa46e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:32.761197 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf57b00-7d07-4114-a53f-e90605ffa46e admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:32.771526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1011/2023] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:32 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:32.775670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf57b00-7d07-4114-a53f-e90605ffa46e admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:32.785930 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf57b00-7d07-4114-a53f-e90605ffa46e admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:32.791017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd39b1a8-b9de-43f6-8984-22f48a7665a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:32.791656 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd39b1a8-b9de-43f6-8984-22f48a7665a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:32.798863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd39b1a8-b9de-43f6-8984-22f48a7665a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:32.798863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd39b1a8-b9de-43f6-8984-22f48a7665a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:32.798863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd39b1a8-b9de-43f6-8984-22f48a7665a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:32.798863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd39b1a8-b9de-43f6-8984-22f48a7665a2 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:32.799204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1013/2024] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:32 2022] POST /identity/v3/projects => generated 373 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:32.800508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffee8faa-35be-42e9-8819-6a566be71e90 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:32.811247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1012/2025] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:32 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:32.812346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd39b1a8-b9de-43f6-8984-22f48a7665a2 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:32.823913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd39b1a8-b9de-43f6-8984-22f48a7665a2 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:32.864760 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffee8faa-35be-42e9-8819-6a566be71e90 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:32.870483 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fee462-77e4-493d-99b4-44b71e172871 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:32.871092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fee462-77e4-493d-99b4-44b71e172871 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:32.871373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fee462-77e4-493d-99b4-44b71e172871 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:32.871639 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fee462-77e4-493d-99b4-44b71e172871 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:32.872294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fee462-77e4-493d-99b4-44b71e172871 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:32.874246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fee462-77e4-493d-99b4-44b71e172871 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:32.883656 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1768b272-6cd5-4913-9bac-8edce841f7c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:32.885429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1768b272-6cd5-4913-9bac-8edce841f7c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:32.885429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1768b272-6cd5-4913-9bac-8edce841f7c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:32.885429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1768b272-6cd5-4913-9bac-8edce841f7c4 admin admin] PATH_INFO: `/v3/projects/ec1cc59360da420492f613c56fb9105e/users/a847012849994b3e9309a5dfb56350b2/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:32.899235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1768b272-6cd5-4913-9bac-8edce841f7c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ec1cc59360da420492f613c56fb9105e, user_id=a847012849994b3e9309a5dfb56350b2, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:32.901944 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1768b272-6cd5-4913-9bac-8edce841f7c4 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:33.037958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1014/2026] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:32 2022] POST /identity/v3/users => generated 287 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:33.041966 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fee462-77e4-493d-99b4-44b71e172871 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:33.053632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fee462-77e4-493d-99b4-44b71e172871 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:33.058459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d656077d-8531-4111-af6e-a94f9b78475a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:33.059061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d656077d-8531-4111-af6e-a94f9b78475a admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:33.059341 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d656077d-8531-4111-af6e-a94f9b78475a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:33.059615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d656077d-8531-4111-af6e-a94f9b78475a admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:33.060316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d656077d-8531-4111-af6e-a94f9b78475a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:33.061921 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d656077d-8531-4111-af6e-a94f9b78475a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:33.063552 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:33.067615 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1013/2027] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:32 2022] PUT /identity/v3/projects/ec1cc59360da420492f613c56fb9105e/users/a847012849994b3e9309a5dfb56350b2/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:33.073046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bd69ec-8ab5-41c4-80bd-0b0cc06fc99b None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:33.073543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bd69ec-8ab5-41c4-80bd-0b0cc06fc99b None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:33.073822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bd69ec-8ab5-41c4-80bd-0b0cc06fc99b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:33.074163 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1015/2028] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:33 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:33.078094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d656077d-8531-4111-af6e-a94f9b78475a admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:33.089653 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d656077d-8531-4111-af6e-a94f9b78475a admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:33.098392 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-56bd69ec-8ab5-41c4-80bd-0b0cc06fc99b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:33.099627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-56bd69ec-8ab5-41c4-80bd-0b0cc06fc99b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:33.134334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:33.177754 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc88601b-f665-4803-aaf2-107642abba9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:33.178377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc88601b-f665-4803-aaf2-107642abba9a admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:33.180214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc88601b-f665-4803-aaf2-107642abba9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:33.180214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc88601b-f665-4803-aaf2-107642abba9a admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:33.180214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc88601b-f665-4803-aaf2-107642abba9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:33.183191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc88601b-f665-4803-aaf2-107642abba9a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:33.193045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1014/2029] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:33.195807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1016/2030] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:33 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:33.200210 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bd69ec-8ab5-41c4-80bd-0b0cc06fc99b None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:33.203924 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc88601b-f665-4803-aaf2-107642abba9a admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:33.206909 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-bc88601b-f665-4803-aaf2-107642abba9a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:33.211721 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bd69ec-8ab5-41c4-80bd-0b0cc06fc99b None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:33.217192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea457f46-0685-4348-b0de-6f84446b48df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:33.217502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea457f46-0685-4348-b0de-6f84446b48df admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:33.217784 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea457f46-0685-4348-b0de-6f84446b48df admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:33.218081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea457f46-0685-4348-b0de-6f84446b48df admin admin] PATH_INFO: `/v3/projects/9c919ad771af41eba22e546d4a6cdd57/users/46556510401945df8b302335ed519e34/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:33.234473 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea457f46-0685-4348-b0de-6f84446b48df admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c919ad771af41eba22e546d4a6cdd57, user_id=46556510401945df8b302335ed519e34, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:33.237913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea457f46-0685-4348-b0de-6f84446b48df admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:33.274791 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc88601b-f665-4803-aaf2-107642abba9a admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:33.279346 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa39ff7-ae49-4f14-ae22-3872b19a9781 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:33.279896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa39ff7-ae49-4f14-ae22-3872b19a9781 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:33.280771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa39ff7-ae49-4f14-ae22-3872b19a9781 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:33.280771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa39ff7-ae49-4f14-ae22-3872b19a9781 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:33.291336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa39ff7-ae49-4f14-ae22-3872b19a9781 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:33.292859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa39ff7-ae49-4f14-ae22-3872b19a9781 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:33.303901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1017/2031] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:26:33.358198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1015/2032] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:33 2022] PUT /identity/v3/projects/9c919ad771af41eba22e546d4a6cdd57/users/46556510401945df8b302335ed519e34/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:33.372303 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6fb441-1c0a-4b4e-92d6-98d2ec63a4d0 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:33.372762 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6fb441-1c0a-4b4e-92d6-98d2ec63a4d0 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:33.373243 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6fb441-1c0a-4b4e-92d6-98d2ec63a4d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:33.404622 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-8e6fb441-1c0a-4b4e-92d6-98d2ec63a4d0 None None] MFA Rules not processed for user `46556510401945df8b302335ed519e34`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:33.405852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-8e6fb441-1c0a-4b4e-92d6-98d2ec63a4d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:33.485928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1018/2033] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:33 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:33.494171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea457f46-0685-4348-b0de-6f84446b48df admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:33.495742 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ea457f46-0685-4348-b0de-6f84446b48df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:33.562795 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea457f46-0685-4348-b0de-6f84446b48df admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:33.566492 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807d009e-711a-42fc-8dca-a32d57d452ec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:33.567097 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807d009e-711a-42fc-8dca-a32d57d452ec service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:33.567420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807d009e-711a-42fc-8dca-a32d57d452ec service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:33.567858 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807d009e-711a-42fc-8dca-a32d57d452ec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:33.576686 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807d009e-711a-42fc-8dca-a32d57d452ec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:33.578165 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807d009e-711a-42fc-8dca-a32d57d452ec service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:33.588466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1016/2034] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:33 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:33.611132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6fb441-1c0a-4b4e-92d6-98d2ec63a4d0 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:33.645032 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:33.683943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6fb441-1c0a-4b4e-92d6-98d2ec63a4d0 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:33.688918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bdf82d-af03-422e-ad71-784b2cb0f3f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:33.689559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bdf82d-af03-422e-ad71-784b2cb0f3f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:33.689928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bdf82d-af03-422e-ad71-784b2cb0f3f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:33.690698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bdf82d-af03-422e-ad71-784b2cb0f3f6 admin admin] PATH_INFO: `/v3/users/46556510401945df8b302335ed519e34` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:33.691862 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bdf82d-af03-422e-ad71-784b2cb0f3f6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=46556510401945df8b302335ed519e34)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:33.693805 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bdf82d-af03-422e-ad71-784b2cb0f3f6 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:33.990456 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-91bdf82d-af03-422e-ad71-784b2cb0f3f6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '46556510401945df8b302335ed519e34'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:33.998237 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-91bdf82d-af03-422e-ad71-784b2cb0f3f6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '46556510401945df8b302335ed519e34'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:34.011790 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-91bdf82d-af03-422e-ad71-784b2cb0f3f6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '46556510401945df8b302335ed519e34'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:34.100036 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-91bdf82d-af03-422e-ad71-784b2cb0f3f6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '46556510401945df8b302335ed519e34'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:34.101029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-91bdf82d-af03-422e-ad71-784b2cb0f3f6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '46556510401945df8b302335ed519e34'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:34.107852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1019/2035] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:33 2022] DELETE /identity/v3/users/46556510401945df8b302335ed519e34 => generated 0 bytes in 497 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:34.113133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aba634-45f8-4d68-b046-a64a46902e72 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:34.113637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aba634-45f8-4d68-b046-a64a46902e72 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:34.113788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aba634-45f8-4d68-b046-a64a46902e72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:34.135702 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-d5aba634-45f8-4d68-b046-a64a46902e72 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:34.137307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d5aba634-45f8-4d68-b046-a64a46902e72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:34.207761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1017/2036] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:34.220015 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bdf82d-af03-422e-ad71-784b2cb0f3f6 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:34.222686 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-91bdf82d-af03-422e-ad71-784b2cb0f3f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:34.293512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bdf82d-af03-422e-ad71-784b2cb0f3f6 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:34.297270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d11e4a-37e2-4e73-a0ed-4c8df50d80b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:34.299613 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d11e4a-37e2-4e73-a0ed-4c8df50d80b0 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:34.299613 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d11e4a-37e2-4e73-a0ed-4c8df50d80b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:34.299613 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d11e4a-37e2-4e73-a0ed-4c8df50d80b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:34.308985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d11e4a-37e2-4e73-a0ed-4c8df50d80b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:34.310534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d11e4a-37e2-4e73-a0ed-4c8df50d80b0 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:34.320697 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1020/2037] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:34.860592 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5aba634-45f8-4d68-b046-a64a46902e72 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:34.860738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d5aba634-45f8-4d68-b046-a64a46902e72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:34.874433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5aba634-45f8-4d68-b046-a64a46902e72 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:34.882183 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af60f54e-f3a4-4e60-97c6-13a7c3a8a450 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:34.882914 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af60f54e-f3a4-4e60-97c6-13a7c3a8a450 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:34.883267 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af60f54e-f3a4-4e60-97c6-13a7c3a8a450 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:34.883549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af60f54e-f3a4-4e60-97c6-13a7c3a8a450 admin admin] PATH_INFO: `/v3/projects/9c919ad771af41eba22e546d4a6cdd57` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:34.885726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af60f54e-f3a4-4e60-97c6-13a7c3a8a450 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9c919ad771af41eba22e546d4a6cdd57)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:34.886856 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af60f54e-f3a4-4e60-97c6-13a7c3a8a450 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:34.930705 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9412d18-812a-4383-8dff-81ec4ef6d2fe None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:34.930992 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9412d18-812a-4383-8dff-81ec4ef6d2fe None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:34.931263 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9412d18-812a-4383-8dff-81ec4ef6d2fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:34.974602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-b9412d18-812a-4383-8dff-81ec4ef6d2fe None None] MFA Rules not processed for user `a847012849994b3e9309a5dfb56350b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:34.974602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-b9412d18-812a-4383-8dff-81ec4ef6d2fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:35.010466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:35.047417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1021/2038] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:34 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:35.051986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9412d18-812a-4383-8dff-81ec4ef6d2fe None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:35.053506 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-b9412d18-812a-4383-8dff-81ec4ef6d2fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:35.066752 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:35.070403 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9412d18-812a-4383-8dff-81ec4ef6d2fe None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:35.071937 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-af60f54e-f3a4-4e60-97c6-13a7c3a8a450 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9c919ad771af41eba22e546d4a6cdd57', 'request_id': 'req-af60f54e-f3a4-4e60-97c6-13a7c3a8a450'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:35.079015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-af60f54e-f3a4-4e60-97c6-13a7c3a8a450 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9c919ad771af41eba22e546d4a6cdd57', 'request_id': 'req-af60f54e-f3a4-4e60-97c6-13a7c3a8a450'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:35.091055 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-af60f54e-f3a4-4e60-97c6-13a7c3a8a450 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9c919ad771af41eba22e546d4a6cdd57', 'request_id': 'req-af60f54e-f3a4-4e60-97c6-13a7c3a8a450'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:35.135156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c0ef78-9ae4-4968-80f4-6439f00deeb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:35.135666 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c0ef78-9ae4-4968-80f4-6439f00deeb9 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:35.135938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c0ef78-9ae4-4968-80f4-6439f00deeb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:35.136237 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c0ef78-9ae4-4968-80f4-6439f00deeb9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:35.136857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c0ef78-9ae4-4968-80f4-6439f00deeb9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:35.138817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c0ef78-9ae4-4968-80f4-6439f00deeb9 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:35.265862 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-af60f54e-f3a4-4e60-97c6-13a7c3a8a450 admin admin] The token cache is being invalidate because project 9c919ad771af41eba22e546d4a6cdd57 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:35.266971 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-af60f54e-f3a4-4e60-97c6-13a7c3a8a450 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:35.266971 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-af60f54e-f3a4-4e60-97c6-13a7c3a8a450 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:35.268871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1018/2039] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:34 2022] DELETE /identity/v3/projects/9c919ad771af41eba22e546d4a6cdd57 => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:35.274566 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c6c193-91f0-4193-a967-fef6b4fbbbc7 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:35.274849 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c6c193-91f0-4193-a967-fef6b4fbbbc7 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:35.275123 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c6c193-91f0-4193-a967-fef6b4fbbbc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:35.292548 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1022/2040] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:35 2022] POST /identity/v3/projects => generated 405 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:35.295989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c0ef78-9ae4-4968-80f4-6439f00deeb9 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:35.297431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-a9c6c193-91f0-4193-a967-fef6b4fbbbc7 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:35.297745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-17c0ef78-9ae4-4968-80f4-6439f00deeb9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:35.298707 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a9c6c193-91f0-4193-a967-fef6b4fbbbc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:35.370887 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1019/2041] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:35.377088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c0ef78-9ae4-4968-80f4-6439f00deeb9 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:35.380311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8f9a96-0d4a-44a7-abd3-b1baca14bd70 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:35.380919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8f9a96-0d4a-44a7-abd3-b1baca14bd70 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:35.381325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8f9a96-0d4a-44a7-abd3-b1baca14bd70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:35.385082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d6b278-f47a-4a85-8d7a-d0b9d8efc797 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:35.387534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d6b278-f47a-4a85-8d7a-d0b9d8efc797 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:35.387534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d6b278-f47a-4a85-8d7a-d0b9d8efc797 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:35.387534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d6b278-f47a-4a85-8d7a-d0b9d8efc797 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:35.387534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d6b278-f47a-4a85-8d7a-d0b9d8efc797 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:35.389168 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d6b278-f47a-4a85-8d7a-d0b9d8efc797 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:35.413968 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-4f8f9a96-0d4a-44a7-abd3-b1baca14bd70 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:35.413968 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-4f8f9a96-0d4a-44a7-abd3-b1baca14bd70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:35.423126 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1020/2042] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:35.428569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8f9a96-0d4a-44a7-abd3-b1baca14bd70 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:35.437782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8f9a96-0d4a-44a7-abd3-b1baca14bd70 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:35.443157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e82628-a7e2-4bc0-b253-d514dfa02a23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:35.443774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e82628-a7e2-4bc0-b253-d514dfa02a23 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:35.444048 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e82628-a7e2-4bc0-b253-d514dfa02a23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:35.444360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e82628-a7e2-4bc0-b253-d514dfa02a23 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:35.445115 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e82628-a7e2-4bc0-b253-d514dfa02a23 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:26:35.445403 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e82628-a7e2-4bc0-b253-d514dfa02a23 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:35.446895 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e82628-a7e2-4bc0-b253-d514dfa02a23 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:35.464057 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1021/2043] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:26:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:26:35.477778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e82628-a7e2-4bc0-b253-d514dfa02a23 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:35.489839 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e82628-a7e2-4bc0-b253-d514dfa02a23 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:35.494542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874b575f-150a-4ba5-8899-bd9a51a77d9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:35.495137 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874b575f-150a-4ba5-8899-bd9a51a77d9a admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:35.495415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874b575f-150a-4ba5-8899-bd9a51a77d9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:35.495687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874b575f-150a-4ba5-8899-bd9a51a77d9a admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:35.496433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874b575f-150a-4ba5-8899-bd9a51a77d9a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:35.498339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874b575f-150a-4ba5-8899-bd9a51a77d9a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:35.512230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1023/2044] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:35 2022] POST /identity/v3/users => generated 309 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:35.514408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d6b278-f47a-4a85-8d7a-d0b9d8efc797 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:35.525695 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d6b278-f47a-4a85-8d7a-d0b9d8efc797 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:35.531205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ded3e7-8dab-4311-8e94-c016c13578d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:35.531856 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ded3e7-8dab-4311-8e94-c016c13578d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:35.532310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ded3e7-8dab-4311-8e94-c016c13578d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:35.532912 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ded3e7-8dab-4311-8e94-c016c13578d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:35.533107 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ded3e7-8dab-4311-8e94-c016c13578d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:35.534738 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ded3e7-8dab-4311-8e94-c016c13578d4 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:35.550741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1024/2045] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:35.555297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ded3e7-8dab-4311-8e94-c016c13578d4 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:35.562636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ded3e7-8dab-4311-8e94-c016c13578d4 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:35.568730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4624d474-2703-4515-aab8-9ca6ca200719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:35.569411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4624d474-2703-4515-aab8-9ca6ca200719 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:35.569690 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4624d474-2703-4515-aab8-9ca6ca200719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:35.570000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4624d474-2703-4515-aab8-9ca6ca200719 admin admin] PATH_INFO: `/v3/projects/4d7e54c4d8ac4a319386310db3cdb835/users/3daa42f052c6497e8334e06806eb192c/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:35.587789 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1022/2046] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:35 2022] POST /identity/v3/projects => generated 373 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:35.589417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874b575f-150a-4ba5-8899-bd9a51a77d9a admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:35.595641 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4624d474-2703-4515-aab8-9ca6ca200719 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d7e54c4d8ac4a319386310db3cdb835, user_id=3daa42f052c6497e8334e06806eb192c, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:35.598384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4624d474-2703-4515-aab8-9ca6ca200719 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:35.659934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874b575f-150a-4ba5-8899-bd9a51a77d9a admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:35.660874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1025/2047] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:35 2022] PUT /identity/v3/projects/4d7e54c4d8ac4a319386310db3cdb835/users/3daa42f052c6497e8334e06806eb192c/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:35.665703 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4624d474-2703-4515-aab8-9ca6ca200719 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:35.758051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5a90e0-4e77-47e6-9534-323ff7b3f782 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:35.758709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5a90e0-4e77-47e6-9534-323ff7b3f782 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:35.759043 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5a90e0-4e77-47e6-9534-323ff7b3f782 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:35.759801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5a90e0-4e77-47e6-9534-323ff7b3f782 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:35.760929 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5a90e0-4e77-47e6-9534-323ff7b3f782 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:35.762881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5a90e0-4e77-47e6-9534-323ff7b3f782 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:35.780944 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4624d474-2703-4515-aab8-9ca6ca200719 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:35.791015 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0785de02-6089-4dda-a4a4-4aa38122530e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:35.791846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0785de02-6089-4dda-a4a4-4aa38122530e admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:35.792321 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0785de02-6089-4dda-a4a4-4aa38122530e admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:35.792597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0785de02-6089-4dda-a4a4-4aa38122530e admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:35.793418 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0785de02-6089-4dda-a4a4-4aa38122530e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:35.795032 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0785de02-6089-4dda-a4a4-4aa38122530e admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:35.808896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1026/2048] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:35 2022] GET /identity/v3/roles => generated 1420 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:35.813689 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0785de02-6089-4dda-a4a4-4aa38122530e admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:35.825302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0785de02-6089-4dda-a4a4-4aa38122530e admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:35.838842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e30e7b-7d9e-424f-a658-6a7102a26b50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:35.838842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e30e7b-7d9e-424f-a658-6a7102a26b50 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:35.838842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e30e7b-7d9e-424f-a658-6a7102a26b50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:35.838842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e30e7b-7d9e-424f-a658-6a7102a26b50 admin admin] PATH_INFO: `/v3/projects/4d7e54c4d8ac4a319386310db3cdb835/users/3daa42f052c6497e8334e06806eb192c/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:35.839305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e30e7b-7d9e-424f-a658-6a7102a26b50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d7e54c4d8ac4a319386310db3cdb835, user_id=3daa42f052c6497e8334e06806eb192c, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:35.839305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e30e7b-7d9e-424f-a658-6a7102a26b50 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:35.853522 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1027/2049] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:35 2022] PUT /identity/v3/projects/4d7e54c4d8ac4a319386310db3cdb835/users/3daa42f052c6497e8334e06806eb192c/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:35.893662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1023/2050] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:35 2022] POST /identity/v3/users => generated 287 bytes in 305 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:35.898133 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e30e7b-7d9e-424f-a658-6a7102a26b50 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:35.971708 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e30e7b-7d9e-424f-a658-6a7102a26b50 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:35.976585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db021a9-7b02-4971-8505-2253b475792c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:35.977189 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db021a9-7b02-4971-8505-2253b475792c admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:35.977464 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db021a9-7b02-4971-8505-2253b475792c admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:35.977731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db021a9-7b02-4971-8505-2253b475792c admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:35.978374 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db021a9-7b02-4971-8505-2253b475792c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:35.979994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db021a9-7b02-4971-8505-2253b475792c admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:35.993662 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1028/2051] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:35 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:35.998077 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5a90e0-4e77-47e6-9534-323ff7b3f782 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:36.009441 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5a90e0-4e77-47e6-9534-323ff7b3f782 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:36.014288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5311ff-d1d8-45f1-82d2-2e676f66d0f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:36.014908 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5311ff-d1d8-45f1-82d2-2e676f66d0f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:36.015179 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5311ff-d1d8-45f1-82d2-2e676f66d0f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:36.015444 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5311ff-d1d8-45f1-82d2-2e676f66d0f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:36.016093 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5311ff-d1d8-45f1-82d2-2e676f66d0f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:36.018215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5311ff-d1d8-45f1-82d2-2e676f66d0f2 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:36.028871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1024/2052] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:35 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:36.040503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db021a9-7b02-4971-8505-2253b475792c admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:36.050886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db021a9-7b02-4971-8505-2253b475792c admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:36.055974 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f08381a-c543-42a2-b051-f4201ec542cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:36.056753 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f08381a-c543-42a2-b051-f4201ec542cd admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:36.057149 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f08381a-c543-42a2-b051-f4201ec542cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:36.057550 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f08381a-c543-42a2-b051-f4201ec542cd admin admin] PATH_INFO: `/v3/projects/a4da17eecf2446e1b119830484bb73c4/users/eef1995bcca54c52ac8df19c60018836/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:36.080755 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f08381a-c543-42a2-b051-f4201ec542cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=a4da17eecf2446e1b119830484bb73c4, user_id=eef1995bcca54c52ac8df19c60018836, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:36.083603 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f08381a-c543-42a2-b051-f4201ec542cd admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:36.196260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1029/2053] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:36 2022] PUT /identity/v3/projects/a4da17eecf2446e1b119830484bb73c4/users/eef1995bcca54c52ac8df19c60018836/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:36.207704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e3f338-8561-4a49-a662-8a105aa860b7 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:36.208245 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e3f338-8561-4a49-a662-8a105aa860b7 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:36.208400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e3f338-8561-4a49-a662-8a105aa860b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:36.239787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f3e3f338-8561-4a49-a662-8a105aa860b7 None None] MFA Rules not processed for user `eef1995bcca54c52ac8df19c60018836`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:36.241007 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f3e3f338-8561-4a49-a662-8a105aa860b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:36.315672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1025/2054] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:36 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:36.327349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f08381a-c543-42a2-b051-f4201ec542cd admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:36.329252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-0f08381a-c543-42a2-b051-f4201ec542cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:36.336941 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:36.411912 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f08381a-c543-42a2-b051-f4201ec542cd admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:36.415997 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed404c0-02ee-4d6e-98df-efaa56340aae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:36.416748 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed404c0-02ee-4d6e-98df-efaa56340aae service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:36.417195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed404c0-02ee-4d6e-98df-efaa56340aae service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:36.417676 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed404c0-02ee-4d6e-98df-efaa56340aae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:36.429871 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed404c0-02ee-4d6e-98df-efaa56340aae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:36.431326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed404c0-02ee-4d6e-98df-efaa56340aae service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:36.446408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1030/2055] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:36 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:36.604535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:37.376531 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e709dad-ce57-40db-acc3-c94732680459 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:37.376531 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e709dad-ce57-40db-acc3-c94732680459 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:37.377641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e709dad-ce57-40db-acc3-c94732680459 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:37.415788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-4e709dad-ce57-40db-acc3-c94732680459 None None] MFA Rules not processed for user `3daa42f052c6497e8334e06806eb192c`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:37.417289 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-4e709dad-ce57-40db-acc3-c94732680459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:37.519658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1026/2056] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:37 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:38.239159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c85d741-7c6c-4d24-9761-e3b3593fe1b6 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:38.239726 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c85d741-7c6c-4d24-9761-e3b3593fe1b6 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:38.239726 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c85d741-7c6c-4d24-9761-e3b3593fe1b6 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:38.241143 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1031/2057] 10.222.0.47 () {54 vars in 921 bytes} [Sun Aug 21 01:26:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:26:38.258307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b49a4d-ef9b-42b7-873f-ba6c8747837e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:38.259082 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b49a4d-ef9b-42b7-873f-ba6c8747837e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:38.259529 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b49a4d-ef9b-42b7-873f-ba6c8747837e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:38.283676 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-07b49a4d-ef9b-42b7-873f-ba6c8747837e None None] MFA Rules not processed for user `972b0f1dbfef4d85a6dfc4c5fd839bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:38.285141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-07b49a4d-ef9b-42b7-873f-ba6c8747837e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:38.331905 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:38.382399 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1027/2058] 10.222.0.47 () {58 vars in 1018 bytes} [Sun Aug 21 01:26:38 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:38.394486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c85d741-7c6c-4d24-9761-e3b3593fe1b6 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:38.396594 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9c85d741-7c6c-4d24-9761-e3b3593fe1b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:38.415852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c85d741-7c6c-4d24-9761-e3b3593fe1b6 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:38.421163 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45173e1-eaf2-4b3a-bc65-618d6b5191f1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '972b0f1dbfef4d85a6dfc4c5fd839bde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:38.421781 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45173e1-eaf2-4b3a-bc65-618d6b5191f1 service placement] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:38.422061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45173e1-eaf2-4b3a-bc65-618d6b5191f1 service placement] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:38.422331 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45173e1-eaf2-4b3a-bc65-618d6b5191f1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:38.432402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45173e1-eaf2-4b3a-bc65-618d6b5191f1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:38.433994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45173e1-eaf2-4b3a-bc65-618d6b5191f1 service placement] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:38.440963 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:38.448753 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1032/2059] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:38 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:39.202050 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b49a4d-ef9b-42b7-873f-ba6c8747837e None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:39.266168 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b49a4d-ef9b-42b7-873f-ba6c8747837e None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:39.271592 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed29dc0-dc0f-49e4-81d9-a125a2d34c38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:39.272282 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed29dc0-dc0f-49e4-81d9-a125a2d34c38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:39.272592 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed29dc0-dc0f-49e4-81d9-a125a2d34c38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:39.272884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed29dc0-dc0f-49e4-81d9-a125a2d34c38 admin admin] PATH_INFO: `/v3/users/a847012849994b3e9309a5dfb56350b2` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:39.274020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed29dc0-dc0f-49e4-81d9-a125a2d34c38 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a847012849994b3e9309a5dfb56350b2)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:39.276019 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed29dc0-dc0f-49e4-81d9-a125a2d34c38 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:39.512756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-8ed29dc0-dc0f-49e4-81d9-a125a2d34c38 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a847012849994b3e9309a5dfb56350b2'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:39.525299 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-8ed29dc0-dc0f-49e4-81d9-a125a2d34c38 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a847012849994b3e9309a5dfb56350b2'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:39.525985 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-8ed29dc0-dc0f-49e4-81d9-a125a2d34c38 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a847012849994b3e9309a5dfb56350b2'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:39.539908 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-8ed29dc0-dc0f-49e4-81d9-a125a2d34c38 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a847012849994b3e9309a5dfb56350b2'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:39.617709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-8ed29dc0-dc0f-49e4-81d9-a125a2d34c38 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a847012849994b3e9309a5dfb56350b2'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:39.629526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1028/2060] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:39 2022] DELETE /identity/v3/users/a847012849994b3e9309a5dfb56350b2 => generated 0 bytes in 428 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:40.003013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45173e1-eaf2-4b3a-bc65-618d6b5191f1 service placement] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:40.018610 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45173e1-eaf2-4b3a-bc65-618d6b5191f1 service placement] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:40.083888 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750f0d79-1068-4eaf-8dee-c8d434b712ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:40.084548 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750f0d79-1068-4eaf-8dee-c8d434b712ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:40.084825 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750f0d79-1068-4eaf-8dee-c8d434b712ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:40.085096 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750f0d79-1068-4eaf-8dee-c8d434b712ff admin admin] PATH_INFO: `/v3/projects/ec1cc59360da420492f613c56fb9105e` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:40.086584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750f0d79-1068-4eaf-8dee-c8d434b712ff admin admin] RBAC: Authorizing `identity:delete_project(project_id=ec1cc59360da420492f613c56fb9105e)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:40.088621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750f0d79-1068-4eaf-8dee-c8d434b712ff admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:40.213901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-750f0d79-1068-4eaf-8dee-c8d434b712ff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ec1cc59360da420492f613c56fb9105e', 'request_id': 'req-750f0d79-1068-4eaf-8dee-c8d434b712ff'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:40.274274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-750f0d79-1068-4eaf-8dee-c8d434b712ff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ec1cc59360da420492f613c56fb9105e', 'request_id': 'req-750f0d79-1068-4eaf-8dee-c8d434b712ff'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:40.282863 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-750f0d79-1068-4eaf-8dee-c8d434b712ff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ec1cc59360da420492f613c56fb9105e', 'request_id': 'req-750f0d79-1068-4eaf-8dee-c8d434b712ff'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:40.296434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-750f0d79-1068-4eaf-8dee-c8d434b712ff admin admin] The token cache is being invalidate because project ec1cc59360da420492f613c56fb9105e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:40.296727 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-750f0d79-1068-4eaf-8dee-c8d434b712ff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:40.297420 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-750f0d79-1068-4eaf-8dee-c8d434b712ff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:40.298942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1033/2061] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:40 2022] DELETE /identity/v3/projects/ec1cc59360da420492f613c56fb9105e => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:40.303510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed29dc0-dc0f-49e4-81d9-a125a2d34c38 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:40.305049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8ed29dc0-dc0f-49e4-81d9-a125a2d34c38 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:40.319399 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed29dc0-dc0f-49e4-81d9-a125a2d34c38 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:40.368670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:40.382939 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea562b0-d1c3-4848-869c-495308162a03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:40.383639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea562b0-d1c3-4848-869c-495308162a03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:40.383918 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea562b0-d1c3-4848-869c-495308162a03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:40.384248 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea562b0-d1c3-4848-869c-495308162a03 admin admin] PATH_INFO: `/v3/users/3daa42f052c6497e8334e06806eb192c` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:40.385373 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea562b0-d1c3-4848-869c-495308162a03 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3daa42f052c6497e8334e06806eb192c)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:40.387334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea562b0-d1c3-4848-869c-495308162a03 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:40.518151 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750f0d79-1068-4eaf-8dee-c8d434b712ff admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:40.519736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-750f0d79-1068-4eaf-8dee-c8d434b712ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:40.539392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750f0d79-1068-4eaf-8dee-c8d434b712ff admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:40.544768 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102d7ef7-b245-49bc-ad0f-2b2f180f82d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:40.548582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102d7ef7-b245-49bc-ad0f-2b2f180f82d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:40.548914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102d7ef7-b245-49bc-ad0f-2b2f180f82d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:40.555055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102d7ef7-b245-49bc-ad0f-2b2f180f82d7 admin admin] PATH_INFO: `/v3/users/b1f05b8f41b440009c7a2ad6b698b758` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:40.555055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102d7ef7-b245-49bc-ad0f-2b2f180f82d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b1f05b8f41b440009c7a2ad6b698b758)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:40.555055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102d7ef7-b245-49bc-ad0f-2b2f180f82d7 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:40.619081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-dea562b0-d1c3-4848-869c-495308162a03 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3daa42f052c6497e8334e06806eb192c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:40.627168 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-dea562b0-d1c3-4848-869c-495308162a03 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3daa42f052c6497e8334e06806eb192c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:40.628287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-dea562b0-d1c3-4848-869c-495308162a03 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3daa42f052c6497e8334e06806eb192c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:40.640254 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-dea562b0-d1c3-4848-869c-495308162a03 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3daa42f052c6497e8334e06806eb192c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:40.699476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-dea562b0-d1c3-4848-869c-495308162a03 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3daa42f052c6497e8334e06806eb192c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:40.706101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1029/2062] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:40 2022] DELETE /identity/v3/users/3daa42f052c6497e8334e06806eb192c => generated 0 bytes in 403 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:40.747340 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-102d7ef7-b245-49bc-ad0f-2b2f180f82d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b1f05b8f41b440009c7a2ad6b698b758'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:40.755690 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-102d7ef7-b245-49bc-ad0f-2b2f180f82d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b1f05b8f41b440009c7a2ad6b698b758'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:40.770235 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-102d7ef7-b245-49bc-ad0f-2b2f180f82d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b1f05b8f41b440009c7a2ad6b698b758'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:40.863608 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-102d7ef7-b245-49bc-ad0f-2b2f180f82d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b1f05b8f41b440009c7a2ad6b698b758'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:40.863608 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-102d7ef7-b245-49bc-ad0f-2b2f180f82d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b1f05b8f41b440009c7a2ad6b698b758'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:40.874715 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1034/2063] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:40 2022] DELETE /identity/v3/users/b1f05b8f41b440009c7a2ad6b698b758 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:41.017408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea562b0-d1c3-4848-869c-495308162a03 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:41.019558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-dea562b0-d1c3-4848-869c-495308162a03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:41.088024 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea562b0-d1c3-4848-869c-495308162a03 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:41.127712 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102d7ef7-b245-49bc-ad0f-2b2f180f82d7 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:41.130056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-102d7ef7-b245-49bc-ad0f-2b2f180f82d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:41.139168 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:41.151516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1018755a-aae9-4005-b24f-37e51fa9924a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:41.152185 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1018755a-aae9-4005-b24f-37e51fa9924a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:41.152863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1018755a-aae9-4005-b24f-37e51fa9924a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:41.153197 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1018755a-aae9-4005-b24f-37e51fa9924a admin admin] PATH_INFO: `/v3/projects/4d7e54c4d8ac4a319386310db3cdb835` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:41.154703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1018755a-aae9-4005-b24f-37e51fa9924a admin admin] RBAC: Authorizing `identity:delete_project(project_id=4d7e54c4d8ac4a319386310db3cdb835)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:41.156546 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1018755a-aae9-4005-b24f-37e51fa9924a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:41.174472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:41.198823 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102d7ef7-b245-49bc-ad0f-2b2f180f82d7 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:41.204065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211ec8f7-8779-4884-bce8-3b06b23b73e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:41.204745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211ec8f7-8779-4884-bce8-3b06b23b73e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:41.205028 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211ec8f7-8779-4884-bce8-3b06b23b73e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:41.205299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211ec8f7-8779-4884-bce8-3b06b23b73e5 admin admin] PATH_INFO: `/v3/projects/7525c268e881425aaf3b6fa530b85ab3` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:41.206951 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211ec8f7-8779-4884-bce8-3b06b23b73e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7525c268e881425aaf3b6fa530b85ab3)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:41.208979 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211ec8f7-8779-4884-bce8-3b06b23b73e5 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:41.286060 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1018755a-aae9-4005-b24f-37e51fa9924a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4d7e54c4d8ac4a319386310db3cdb835', 'request_id': 'req-1018755a-aae9-4005-b24f-37e51fa9924a'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:41.291217 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1018755a-aae9-4005-b24f-37e51fa9924a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4d7e54c4d8ac4a319386310db3cdb835', 'request_id': 'req-1018755a-aae9-4005-b24f-37e51fa9924a'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:41.297597 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1018755a-aae9-4005-b24f-37e51fa9924a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4d7e54c4d8ac4a319386310db3cdb835', 'request_id': 'req-1018755a-aae9-4005-b24f-37e51fa9924a'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:41.315062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-211ec8f7-8779-4884-bce8-3b06b23b73e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7525c268e881425aaf3b6fa530b85ab3', 'request_id': 'req-211ec8f7-8779-4884-bce8-3b06b23b73e5'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:41.410644 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1018755a-aae9-4005-b24f-37e51fa9924a admin admin] The token cache is being invalidate because project 4d7e54c4d8ac4a319386310db3cdb835 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:41.410935 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1018755a-aae9-4005-b24f-37e51fa9924a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:41.411575 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1018755a-aae9-4005-b24f-37e51fa9924a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:41.413190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1030/2064] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:41 2022] DELETE /identity/v3/projects/4d7e54c4d8ac4a319386310db3cdb835 => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:41.417973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d0ee09-9f32-4a26-9bb2-986ac727362c None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:41.418256 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d0ee09-9f32-4a26-9bb2-986ac727362c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:41.418529 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d0ee09-9f32-4a26-9bb2-986ac727362c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:41.438779 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-64d0ee09-9f32-4a26-9bb2-986ac727362c None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:41.439986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-64d0ee09-9f32-4a26-9bb2-986ac727362c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:41.482163 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-211ec8f7-8779-4884-bce8-3b06b23b73e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7525c268e881425aaf3b6fa530b85ab3', 'request_id': 'req-211ec8f7-8779-4884-bce8-3b06b23b73e5'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:41.488346 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-211ec8f7-8779-4884-bce8-3b06b23b73e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7525c268e881425aaf3b6fa530b85ab3', 'request_id': 'req-211ec8f7-8779-4884-bce8-3b06b23b73e5'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:41.499402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1031/2065] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:41.499973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-211ec8f7-8779-4884-bce8-3b06b23b73e5 admin admin] The token cache is being invalidate because project 7525c268e881425aaf3b6fa530b85ab3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:41.500496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-211ec8f7-8779-4884-bce8-3b06b23b73e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:41.501189 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-211ec8f7-8779-4884-bce8-3b06b23b73e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:41.502647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1035/2066] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:41 2022] DELETE /identity/v3/projects/7525c268e881425aaf3b6fa530b85ab3 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:41.507332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211ec8f7-8779-4884-bce8-3b06b23b73e5 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:41.507571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa64caea-e5cc-4c50-b93b-55b42e380219 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:41.507856 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa64caea-e5cc-4c50-b93b-55b42e380219 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:41.508155 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa64caea-e5cc-4c50-b93b-55b42e380219 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:41.508646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-211ec8f7-8779-4884-bce8-3b06b23b73e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:41.523891 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211ec8f7-8779-4884-bce8-3b06b23b73e5 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:41.528600 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-fa64caea-e5cc-4c50-b93b-55b42e380219 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:41.528987 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712afbae-0b74-4671-904c-7ece2f01e3d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:41.529668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712afbae-0b74-4671-904c-7ece2f01e3d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:41.529806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-fa64caea-e5cc-4c50-b93b-55b42e380219 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:41.529949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712afbae-0b74-4671-904c-7ece2f01e3d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:41.530216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712afbae-0b74-4671-904c-7ece2f01e3d9 admin admin] PATH_INFO: `/v3/users/6462955967a84063b74d7f5b8770601b` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:41.531352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712afbae-0b74-4671-904c-7ece2f01e3d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6462955967a84063b74d7f5b8770601b)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:41.533511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712afbae-0b74-4671-904c-7ece2f01e3d9 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:41.541368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1032/2067] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:41.542255 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa64caea-e5cc-4c50-b93b-55b42e380219 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:41.551272 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa64caea-e5cc-4c50-b93b-55b42e380219 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:41.557451 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2934e6-a086-45d4-ab34-bd1bdbf9a2c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:41.558052 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2934e6-a086-45d4-ab34-bd1bdbf9a2c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:41.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2934e6-a086-45d4-ab34-bd1bdbf9a2c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:41.558618 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2934e6-a086-45d4-ab34-bd1bdbf9a2c5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:41.559226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2934e6-a086-45d4-ab34-bd1bdbf9a2c5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:26:41.559523 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2934e6-a086-45d4-ab34-bd1bdbf9a2c5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:41.561152 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2934e6-a086-45d4-ab34-bd1bdbf9a2c5 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:41.574076 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1033/2068] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:26:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:26:41.578052 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2934e6-a086-45d4-ab34-bd1bdbf9a2c5 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:41.588068 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2934e6-a086-45d4-ab34-bd1bdbf9a2c5 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:41.592943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26986874-7f72-4daf-91a5-05db38bd2eb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:41.593537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26986874-7f72-4daf-91a5-05db38bd2eb6 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:41.593815 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26986874-7f72-4daf-91a5-05db38bd2eb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:41.594085 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26986874-7f72-4daf-91a5-05db38bd2eb6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:41.594694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26986874-7f72-4daf-91a5-05db38bd2eb6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:41.596644 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26986874-7f72-4daf-91a5-05db38bd2eb6 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:41.714917 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1034/2069] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:41 2022] POST /identity/v3/projects => generated 387 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:41.720671 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26986874-7f72-4daf-91a5-05db38bd2eb6 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:41.722558 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-712afbae-0b74-4671-904c-7ece2f01e3d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6462955967a84063b74d7f5b8770601b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:41.730495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-712afbae-0b74-4671-904c-7ece2f01e3d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6462955967a84063b74d7f5b8770601b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:41.743949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-712afbae-0b74-4671-904c-7ece2f01e3d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6462955967a84063b74d7f5b8770601b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:41.789909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26986874-7f72-4daf-91a5-05db38bd2eb6 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:41.798021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f012257a-c3f6-4c0f-8081-0de757d8ffee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:41.798021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f012257a-c3f6-4c0f-8081-0de757d8ffee admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:41.798021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f012257a-c3f6-4c0f-8081-0de757d8ffee admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:41.798021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f012257a-c3f6-4c0f-8081-0de757d8ffee admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:41.798450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f012257a-c3f6-4c0f-8081-0de757d8ffee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:41.798450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f012257a-c3f6-4c0f-8081-0de757d8ffee admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:41.807771 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:41.816719 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-712afbae-0b74-4671-904c-7ece2f01e3d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6462955967a84063b74d7f5b8770601b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:41.817439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-712afbae-0b74-4671-904c-7ece2f01e3d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6462955967a84063b74d7f5b8770601b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:41.824478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1036/2070] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:41 2022] DELETE /identity/v3/users/6462955967a84063b74d7f5b8770601b => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:41.912648 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1035/2071] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:41 2022] POST /identity/v3/users => generated 294 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:41.916336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712afbae-0b74-4671-904c-7ece2f01e3d9 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:41.917877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-712afbae-0b74-4671-904c-7ece2f01e3d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:41.925490 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:41.990111 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712afbae-0b74-4671-904c-7ece2f01e3d9 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:41.995470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a2c847-72dc-4e45-837c-377ad05fc566 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:41.996086 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a2c847-72dc-4e45-837c-377ad05fc566 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:41.996391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a2c847-72dc-4e45-837c-377ad05fc566 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:41.999060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a2c847-72dc-4e45-837c-377ad05fc566 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:41.999060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a2c847-72dc-4e45-837c-377ad05fc566 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:41.999060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a2c847-72dc-4e45-837c-377ad05fc566 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:42.020839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1037/2072] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:41 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:42.026273 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f012257a-c3f6-4c0f-8081-0de757d8ffee admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:42.038278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f012257a-c3f6-4c0f-8081-0de757d8ffee admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:42.043487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a9dd53-72c1-4104-ad73-08e2cfa6f4df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:42.044315 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a9dd53-72c1-4104-ad73-08e2cfa6f4df admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:42.044722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a9dd53-72c1-4104-ad73-08e2cfa6f4df admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:42.044832 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a9dd53-72c1-4104-ad73-08e2cfa6f4df admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:42.045478 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a9dd53-72c1-4104-ad73-08e2cfa6f4df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:42.047323 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a9dd53-72c1-4104-ad73-08e2cfa6f4df admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:42.055452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a2c847-72dc-4e45-837c-377ad05fc566 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:42.057250 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-33a2c847-72dc-4e45-837c-377ad05fc566 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:42.070716 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a2c847-72dc-4e45-837c-377ad05fc566 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:42.075272 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1036/2073] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:42 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:42.075454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399fcac7-cf3a-483d-a31c-98c9150c087d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:42.076082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399fcac7-cf3a-483d-a31c-98c9150c087d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:42.076388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399fcac7-cf3a-483d-a31c-98c9150c087d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:42.076660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399fcac7-cf3a-483d-a31c-98c9150c087d admin admin] PATH_INFO: `/v3/users/eef1995bcca54c52ac8df19c60018836` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:42.077863 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399fcac7-cf3a-483d-a31c-98c9150c087d admin admin] RBAC: Authorizing `identity:delete_user(user_id=eef1995bcca54c52ac8df19c60018836)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:42.079425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a9dd53-72c1-4104-ad73-08e2cfa6f4df admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:42.079806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399fcac7-cf3a-483d-a31c-98c9150c087d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:42.089886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a9dd53-72c1-4104-ad73-08e2cfa6f4df admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:42.094775 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fb70ac-c510-424a-8384-b01ec3947661 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:42.095491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fb70ac-c510-424a-8384-b01ec3947661 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:42.095786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fb70ac-c510-424a-8384-b01ec3947661 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:42.096050 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fb70ac-c510-424a-8384-b01ec3947661 admin admin] PATH_INFO: `/v3/projects/f712e114877b450182add63063b4780d/users/1060173cf2504bf58cdd3b34e525609f/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:42.111682 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fb70ac-c510-424a-8384-b01ec3947661 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f712e114877b450182add63063b4780d, user_id=1060173cf2504bf58cdd3b34e525609f, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:42.114961 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fb70ac-c510-424a-8384-b01ec3947661 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:42.231343 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1037/2074] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:42 2022] PUT /identity/v3/projects/f712e114877b450182add63063b4780d/users/1060173cf2504bf58cdd3b34e525609f/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:42.232211 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fb70ac-c510-424a-8384-b01ec3947661 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:42.233760 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-64fb70ac-c510-424a-8384-b01ec3947661 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:42.297252 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fb70ac-c510-424a-8384-b01ec3947661 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:42.302698 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a065c254-69d9-4874-9720-d0b897296057 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:42.303262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a065c254-69d9-4874-9720-d0b897296057 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:42.303538 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a065c254-69d9-4874-9720-d0b897296057 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:42.303806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a065c254-69d9-4874-9720-d0b897296057 admin admin] PATH_INFO: `/v3/projects/c1299e8a288741779f68680d3e3d07cd` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:42.304946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a065c254-69d9-4874-9720-d0b897296057 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c1299e8a288741779f68680d3e3d07cd)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:42.307034 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a065c254-69d9-4874-9720-d0b897296057 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:42.344195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-399fcac7-cf3a-483d-a31c-98c9150c087d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eef1995bcca54c52ac8df19c60018836'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:42.353360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-399fcac7-cf3a-483d-a31c-98c9150c087d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eef1995bcca54c52ac8df19c60018836'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:42.366290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-399fcac7-cf3a-483d-a31c-98c9150c087d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eef1995bcca54c52ac8df19c60018836'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:42.433497 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-399fcac7-cf3a-483d-a31c-98c9150c087d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eef1995bcca54c52ac8df19c60018836'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:42.434116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-399fcac7-cf3a-483d-a31c-98c9150c087d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eef1995bcca54c52ac8df19c60018836'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:42.440734 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1038/2075] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:42 2022] DELETE /identity/v3/users/eef1995bcca54c52ac8df19c60018836 => generated 0 bytes in 386 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:42.441956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f69ca9c-2b70-4247-a8ac-e0a6ab5eaf06 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:42.442240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f69ca9c-2b70-4247-a8ac-e0a6ab5eaf06 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:42.442692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f69ca9c-2b70-4247-a8ac-e0a6ab5eaf06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:42.451260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a065c254-69d9-4874-9720-d0b897296057 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c1299e8a288741779f68680d3e3d07cd', 'request_id': 'req-a065c254-69d9-4874-9720-d0b897296057'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:42.457332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a065c254-69d9-4874-9720-d0b897296057 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c1299e8a288741779f68680d3e3d07cd', 'request_id': 'req-a065c254-69d9-4874-9720-d0b897296057'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:42.463786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a065c254-69d9-4874-9720-d0b897296057 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c1299e8a288741779f68680d3e3d07cd', 'request_id': 'req-a065c254-69d9-4874-9720-d0b897296057'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:42.475660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-7f69ca9c-2b70-4247-a8ac-e0a6ab5eaf06 None None] MFA Rules not processed for user `1060173cf2504bf58cdd3b34e525609f`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:42.476932 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-7f69ca9c-2b70-4247-a8ac-e0a6ab5eaf06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:42.540308 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1039/2076] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:42 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:42.541788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e76556a-d773-4a4a-bd98-51adc91f5af9 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:42.542070 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e76556a-d773-4a4a-bd98-51adc91f5af9 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:42.542354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e76556a-d773-4a4a-bd98-51adc91f5af9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:42.545716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a065c254-69d9-4874-9720-d0b897296057 admin admin] The token cache is being invalidate because project c1299e8a288741779f68680d3e3d07cd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:42.546313 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a065c254-69d9-4874-9720-d0b897296057 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:42.546990 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a065c254-69d9-4874-9720-d0b897296057 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:42.552587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1038/2077] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:42 2022] DELETE /identity/v3/projects/c1299e8a288741779f68680d3e3d07cd => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:42.552587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a065c254-69d9-4874-9720-d0b897296057 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:42.555547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a065c254-69d9-4874-9720-d0b897296057 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:42.564299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-8e76556a-d773-4a4a-bd98-51adc91f5af9 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:42.565556 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-8e76556a-d773-4a4a-bd98-51adc91f5af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:42.625042 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:INSERT=4,keystone:SELECT=89,keystone:DELETE=7 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:42.632427 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1040/2078] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:42.633746 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b2662a-d34c-434a-af87-7df4d8eb0ab7 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:42.634035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b2662a-d34c-434a-af87-7df4d8eb0ab7 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:42.634447 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b2662a-d34c-434a-af87-7df4d8eb0ab7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:42.644403 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a065c254-69d9-4874-9720-d0b897296057 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:42.649620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de67b83-5fb0-4780-865c-87cb9332284a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:42.650222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de67b83-5fb0-4780-865c-87cb9332284a admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:42.650496 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de67b83-5fb0-4780-865c-87cb9332284a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:42.650763 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de67b83-5fb0-4780-865c-87cb9332284a admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:42.651377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de67b83-5fb0-4780-865c-87cb9332284a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:42.653346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de67b83-5fb0-4780-865c-87cb9332284a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:42.656965 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-c9b2662a-d34c-434a-af87-7df4d8eb0ab7 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:42.658167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c9b2662a-d34c-434a-af87-7df4d8eb0ab7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:42.668536 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1041/2079] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:42.670115 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b2662a-d34c-434a-af87-7df4d8eb0ab7 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:42.671464 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c9b2662a-d34c-434a-af87-7df4d8eb0ab7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:42.726396 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b2662a-d34c-434a-af87-7df4d8eb0ab7 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:42.730331 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2f28d3-6563-4f61-a0f4-667d8315a09d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:42.730966 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2f28d3-6563-4f61-a0f4-667d8315a09d service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:42.731261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2f28d3-6563-4f61-a0f4-667d8315a09d service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:42.731619 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2f28d3-6563-4f61-a0f4-667d8315a09d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:42.731902 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1039/2080] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:42 2022] POST /identity/v3/projects => generated 381 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:42.733268 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37ed28c-925a-465b-9467-9f3daea64ae4 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:42.733548 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37ed28c-925a-465b-9467-9f3daea64ae4 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:42.733819 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37ed28c-925a-465b-9467-9f3daea64ae4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:42.745305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2f28d3-6563-4f61-a0f4-667d8315a09d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:42.746786 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2f28d3-6563-4f61-a0f4-667d8315a09d service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:42.755008 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-d37ed28c-925a-465b-9467-9f3daea64ae4 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:42.756351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d37ed28c-925a-465b-9467-9f3daea64ae4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:42.757918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1042/2081] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:42.764333 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2f28d3-6563-4f61-a0f4-667d8315a09d service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:42.775141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2f28d3-6563-4f61-a0f4-667d8315a09d service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:42.821736 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1040/2082] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:42.826226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37ed28c-925a-465b-9467-9f3daea64ae4 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:42.836042 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37ed28c-925a-465b-9467-9f3daea64ae4 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:42.839459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b8f3a9-3860-4661-a70e-aaa41ea906fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:42.840064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b8f3a9-3860-4661-a70e-aaa41ea906fc admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:42.844196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b8f3a9-3860-4661-a70e-aaa41ea906fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:42.844196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b8f3a9-3860-4661-a70e-aaa41ea906fc admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:42.844380 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92ca7a2-7b6d-4cca-a9b2-6931ffcbee67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:42.844483 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b8f3a9-3860-4661-a70e-aaa41ea906fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:42.844568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92ca7a2-7b6d-4cca-a9b2-6931ffcbee67 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:42.844568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92ca7a2-7b6d-4cca-a9b2-6931ffcbee67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:42.844568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92ca7a2-7b6d-4cca-a9b2-6931ffcbee67 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:42.844568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92ca7a2-7b6d-4cca-a9b2-6931ffcbee67 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:26:42.844568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92ca7a2-7b6d-4cca-a9b2-6931ffcbee67 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:42.844942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b8f3a9-3860-4661-a70e-aaa41ea906fc admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:42.845053 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92ca7a2-7b6d-4cca-a9b2-6931ffcbee67 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:42.856860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1041/2083] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:26:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:26:42.861355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92ca7a2-7b6d-4cca-a9b2-6931ffcbee67 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:42.872653 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92ca7a2-7b6d-4cca-a9b2-6931ffcbee67 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:42.878079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e933abba-0468-471e-b930-56ff9d5d1579 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:42.878764 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e933abba-0468-471e-b930-56ff9d5d1579 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:42.879042 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e933abba-0468-471e-b930-56ff9d5d1579 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:42.879321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e933abba-0468-471e-b930-56ff9d5d1579 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:42.879949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e933abba-0468-471e-b930-56ff9d5d1579 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:42.881878 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e933abba-0468-471e-b930-56ff9d5d1579 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:42.967736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1043/2084] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:42 2022] POST /identity/v3/users => generated 297 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:42.969258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b8f3a9-3860-4661-a70e-aaa41ea906fc admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:42.979324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b8f3a9-3860-4661-a70e-aaa41ea906fc admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:42.984082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a41dda-2e04-4664-81d2-bf37eb60fefa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:42.984693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a41dda-2e04-4664-81d2-bf37eb60fefa admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:42.984970 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a41dda-2e04-4664-81d2-bf37eb60fefa admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:42.985238 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a41dda-2e04-4664-81d2-bf37eb60fefa admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:42.985869 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a41dda-2e04-4664-81d2-bf37eb60fefa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:42.987486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a41dda-2e04-4664-81d2-bf37eb60fefa admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:42.991597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:42.999541 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1042/2085] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:42 2022] POST /identity/v3/projects => generated 375 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:43.001666 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1044/2086] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:42 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:43.006764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a41dda-2e04-4664-81d2-bf37eb60fefa admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:43.006940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e933abba-0468-471e-b930-56ff9d5d1579 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:43.017775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a41dda-2e04-4664-81d2-bf37eb60fefa admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:43.087185 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca1e5de-1ce3-4941-a98f-1cf78b96d7cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:43.087899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca1e5de-1ce3-4941-a98f-1cf78b96d7cb admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:43.088212 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca1e5de-1ce3-4941-a98f-1cf78b96d7cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:43.088489 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca1e5de-1ce3-4941-a98f-1cf78b96d7cb admin admin] PATH_INFO: `/v3/projects/02fb285c7092440181b91bb18f25e993/users/654edc3fba324776bb66e300e22ac5e8/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:43.105868 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e933abba-0468-471e-b930-56ff9d5d1579 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:43.112640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca1e5de-1ce3-4941-a98f-1cf78b96d7cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=02fb285c7092440181b91bb18f25e993, user_id=654edc3fba324776bb66e300e22ac5e8, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:43.112640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca1e5de-1ce3-4941-a98f-1cf78b96d7cb admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:43.114572 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff80cc2-dfb5-4b0e-b38e-763ce318545b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:43.115307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff80cc2-dfb5-4b0e-b38e-763ce318545b admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:43.115662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff80cc2-dfb5-4b0e-b38e-763ce318545b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:43.116003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff80cc2-dfb5-4b0e-b38e-763ce318545b admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:43.116882 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff80cc2-dfb5-4b0e-b38e-763ce318545b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:43.119354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff80cc2-dfb5-4b0e-b38e-763ce318545b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:43.211365 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1045/2087] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:43 2022] PUT /identity/v3/projects/02fb285c7092440181b91bb18f25e993/users/654edc3fba324776bb66e300e22ac5e8/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:43.215375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca1e5de-1ce3-4941-a98f-1cf78b96d7cb admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:43.225945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1043/2088] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:43 2022] POST /identity/v3/users => generated 288 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:43.229828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff80cc2-dfb5-4b0e-b38e-763ce318545b admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:43.240325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff80cc2-dfb5-4b0e-b38e-763ce318545b admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:43.301622 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca1e5de-1ce3-4941-a98f-1cf78b96d7cb admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:43.306378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7eb018-f551-4c49-95fa-2c080c5a6643 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:43.306976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7eb018-f551-4c49-95fa-2c080c5a6643 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:43.307247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7eb018-f551-4c49-95fa-2c080c5a6643 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:43.307517 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7eb018-f551-4c49-95fa-2c080c5a6643 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:43.308191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7eb018-f551-4c49-95fa-2c080c5a6643 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:43.308304 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33956ecb-624d-47f4-a52e-29b850ce603a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:43.308906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33956ecb-624d-47f4-a52e-29b850ce603a admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:43.309187 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33956ecb-624d-47f4-a52e-29b850ce603a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:43.309491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33956ecb-624d-47f4-a52e-29b850ce603a admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:43.309781 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7eb018-f551-4c49-95fa-2c080c5a6643 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:43.310160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33956ecb-624d-47f4-a52e-29b850ce603a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:43.311829 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33956ecb-624d-47f4-a52e-29b850ce603a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:43.324775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1046/2089] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:43 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:43.325014 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1044/2090] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:43 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:43.328850 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7eb018-f551-4c49-95fa-2c080c5a6643 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:43.328979 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33956ecb-624d-47f4-a52e-29b850ce603a admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:43.336537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:43.339981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7eb018-f551-4c49-95fa-2c080c5a6643 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:43.341723 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33956ecb-624d-47f4-a52e-29b850ce603a admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:43.347762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b1206b-1308-40c5-b618-ff3ddea71eba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:43.352358 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b1206b-1308-40c5-b618-ff3ddea71eba admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:43.352358 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b1206b-1308-40c5-b618-ff3ddea71eba admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:43.352358 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b1206b-1308-40c5-b618-ff3ddea71eba admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:43.352358 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b1206b-1308-40c5-b618-ff3ddea71eba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:43.352358 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b1206b-1308-40c5-b618-ff3ddea71eba admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:43.353313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39dbccc-2cdd-4a46-b0f8-d4a6c21c657b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:43.354521 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39dbccc-2cdd-4a46-b0f8-d4a6c21c657b admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:43.354795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39dbccc-2cdd-4a46-b0f8-d4a6c21c657b admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:43.355065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39dbccc-2cdd-4a46-b0f8-d4a6c21c657b admin admin] PATH_INFO: `/v3/projects/02fb285c7092440181b91bb18f25e993/users/654edc3fba324776bb66e300e22ac5e8/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:43.358563 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39dbccc-2cdd-4a46-b0f8-d4a6c21c657b admin admin] RBAC: Authorizing `identity:create_grant(project_id=02fb285c7092440181b91bb18f25e993, user_id=654edc3fba324776bb66e300e22ac5e8, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:43.362192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39dbccc-2cdd-4a46-b0f8-d4a6c21c657b admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:43.373731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1045/2091] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:43 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:43.373731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b1206b-1308-40c5-b618-ff3ddea71eba admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:43.373731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-15b1206b-1308-40c5-b618-ff3ddea71eba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:43.392160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1047/2092] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:43 2022] PUT /identity/v3/projects/02fb285c7092440181b91bb18f25e993/users/654edc3fba324776bb66e300e22ac5e8/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:43.392160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39dbccc-2cdd-4a46-b0f8-d4a6c21c657b admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:43.392921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b1206b-1308-40c5-b618-ff3ddea71eba admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:43.450866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1f554f-c257-45cd-b729-139420d1a295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:43.451506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1f554f-c257-45cd-b729-139420d1a295 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:43.451952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1f554f-c257-45cd-b729-139420d1a295 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:43.452271 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1f554f-c257-45cd-b729-139420d1a295 admin admin] PATH_INFO: `/v3/projects/a4da17eecf2446e1b119830484bb73c4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:43.454282 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1f554f-c257-45cd-b729-139420d1a295 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a4da17eecf2446e1b119830484bb73c4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:43.456235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1f554f-c257-45cd-b729-139420d1a295 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:43.464877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39dbccc-2cdd-4a46-b0f8-d4a6c21c657b admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:43.470033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736138de-1e70-4ba8-8859-a508f5f0f2b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:43.470782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736138de-1e70-4ba8-8859-a508f5f0f2b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:43.471062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736138de-1e70-4ba8-8859-a508f5f0f2b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:43.471334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736138de-1e70-4ba8-8859-a508f5f0f2b6 admin admin] PATH_INFO: `/v3/projects/17245d3d23d1456fb1a2936887a75dee/users/27504b8a119640b390c27decbfb72da0/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:43.486168 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736138de-1e70-4ba8-8859-a508f5f0f2b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=17245d3d23d1456fb1a2936887a75dee, user_id=27504b8a119640b390c27decbfb72da0, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:43.489559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736138de-1e70-4ba8-8859-a508f5f0f2b6 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:43.519236 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:43.579984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1048/2093] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:43 2022] PUT /identity/v3/projects/17245d3d23d1456fb1a2936887a75dee/users/27504b8a119640b390c27decbfb72da0/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:43.581280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034ac5aa-3a65-4969-9ee3-c46f61cd2446 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:43.581572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034ac5aa-3a65-4969-9ee3-c46f61cd2446 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:43.581848 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034ac5aa-3a65-4969-9ee3-c46f61cd2446 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:43.588868 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4c1f554f-c257-45cd-b729-139420d1a295 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a4da17eecf2446e1b119830484bb73c4', 'request_id': 'req-4c1f554f-c257-45cd-b729-139420d1a295'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:43.596569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4c1f554f-c257-45cd-b729-139420d1a295 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a4da17eecf2446e1b119830484bb73c4', 'request_id': 'req-4c1f554f-c257-45cd-b729-139420d1a295'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:43.605494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4c1f554f-c257-45cd-b729-139420d1a295 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a4da17eecf2446e1b119830484bb73c4', 'request_id': 'req-4c1f554f-c257-45cd-b729-139420d1a295'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:43.614674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-034ac5aa-3a65-4969-9ee3-c46f61cd2446 None None] MFA Rules not processed for user `654edc3fba324776bb66e300e22ac5e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:43.615709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-034ac5aa-3a65-4969-9ee3-c46f61cd2446 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:43.687534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1049/2094] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:43 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:43.689237 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1198b0d9-18f9-4eb5-bfce-53d9d3a42f75 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:43.689655 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1198b0d9-18f9-4eb5-bfce-53d9d3a42f75 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:43.689971 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1198b0d9-18f9-4eb5-bfce-53d9d3a42f75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:43.698308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4c1f554f-c257-45cd-b729-139420d1a295 admin admin] The token cache is being invalidate because project a4da17eecf2446e1b119830484bb73c4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:43.698659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4c1f554f-c257-45cd-b729-139420d1a295 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:43.699484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-4c1f554f-c257-45cd-b729-139420d1a295 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:43.704952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1046/2095] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:43 2022] DELETE /identity/v3/projects/a4da17eecf2446e1b119830484bb73c4 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:43.704952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1f554f-c257-45cd-b729-139420d1a295 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:43.704952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-4c1f554f-c257-45cd-b729-139420d1a295 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:43.726013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-1198b0d9-18f9-4eb5-bfce-53d9d3a42f75 None None] MFA Rules not processed for user `27504b8a119640b390c27decbfb72da0`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:43.727493 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1198b0d9-18f9-4eb5-bfce-53d9d3a42f75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:43.785523 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:43.797716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1f554f-c257-45cd-b729-139420d1a295 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:43.805447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01af7ab8-46b2-4441-8448-5ad3dcd1a72d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:43.806117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01af7ab8-46b2-4441-8448-5ad3dcd1a72d service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:43.806405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01af7ab8-46b2-4441-8448-5ad3dcd1a72d service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:43.806810 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01af7ab8-46b2-4441-8448-5ad3dcd1a72d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:43.808808 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-01af7ab8-46b2-4441-8448-5ad3dcd1a72d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:43.826325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01af7ab8-46b2-4441-8448-5ad3dcd1a72d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:43.828300 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01af7ab8-46b2-4441-8448-5ad3dcd1a72d service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:43.828622 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1050/2096] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:43 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:43.829902 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4090cb86-917d-46e9-9d27-b8e87fc0c3ab None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:43.830243 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4090cb86-917d-46e9-9d27-b8e87fc0c3ab None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:43.830462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4090cb86-917d-46e9-9d27-b8e87fc0c3ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:43.836670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:43.843647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1047/2097] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:43 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:43.857942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-4090cb86-917d-46e9-9d27-b8e87fc0c3ab None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:43.858633 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-4090cb86-917d-46e9-9d27-b8e87fc0c3ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:43.874187 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01af7ab8-46b2-4441-8448-5ad3dcd1a72d service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:43.884903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01af7ab8-46b2-4441-8448-5ad3dcd1a72d service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:43.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a16104-f69e-41ca-bf45-d5cf3c98208d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:43.889728 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a16104-f69e-41ca-bf45-d5cf3c98208d service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:43.890002 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a16104-f69e-41ca-bf45-d5cf3c98208d service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:43.890277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a16104-f69e-41ca-bf45-d5cf3c98208d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:43.901031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a16104-f69e-41ca-bf45-d5cf3c98208d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:43.906220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a16104-f69e-41ca-bf45-d5cf3c98208d service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:43.918894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1048/2098] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:43 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:43.944150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1051/2099] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:43.947006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcc05a1-ada9-4d2e-b402-6e6304d85c9f None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:43.947417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcc05a1-ada9-4d2e-b402-6e6304d85c9f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:43.947800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcc05a1-ada9-4d2e-b402-6e6304d85c9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:43.972195 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-cdcc05a1-ada9-4d2e-b402-6e6304d85c9f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:43.973860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-cdcc05a1-ada9-4d2e-b402-6e6304d85c9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:43.991060 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1049/2100] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:43.997775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4090cb86-917d-46e9-9d27-b8e87fc0c3ab None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:44.009981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4090cb86-917d-46e9-9d27-b8e87fc0c3ab None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:44.016378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d6635c-9101-493a-9e31-e19fc9a2f183 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:44.017036 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d6635c-9101-493a-9e31-e19fc9a2f183 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:44.017366 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d6635c-9101-493a-9e31-e19fc9a2f183 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:44.017651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d6635c-9101-493a-9e31-e19fc9a2f183 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:44.018303 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d6635c-9101-493a-9e31-e19fc9a2f183 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:26:44.018667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d6635c-9101-493a-9e31-e19fc9a2f183 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:44.021533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d6635c-9101-493a-9e31-e19fc9a2f183 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:44.041816 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1052/2101] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:26:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:26:44.047877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcc05a1-ada9-4d2e-b402-6e6304d85c9f None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:44.059583 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcc05a1-ada9-4d2e-b402-6e6304d85c9f None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:44.064126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d6635c-9101-493a-9e31-e19fc9a2f183 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:44.066688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c3d6635c-9101-493a-9e31-e19fc9a2f183 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:44.071747 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea5e01b-2389-428f-917f-eabcc9d2c74e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:44.072502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea5e01b-2389-428f-917f-eabcc9d2c74e admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:44.072774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea5e01b-2389-428f-917f-eabcc9d2c74e admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:44.078489 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea5e01b-2389-428f-917f-eabcc9d2c74e admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:44.078489 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea5e01b-2389-428f-917f-eabcc9d2c74e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:44.078489 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea5e01b-2389-428f-917f-eabcc9d2c74e admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:44.081406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d6635c-9101-493a-9e31-e19fc9a2f183 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:44.087819 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6a1116-39f1-4cf5-9180-b5077251da9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:44.088537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6a1116-39f1-4cf5-9180-b5077251da9a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:44.088834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6a1116-39f1-4cf5-9180-b5077251da9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:44.089092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6a1116-39f1-4cf5-9180-b5077251da9a admin admin] PATH_INFO: `/v3/users/1060173cf2504bf58cdd3b34e525609f` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:44.090266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6a1116-39f1-4cf5-9180-b5077251da9a admin admin] RBAC: Authorizing `identity:delete_user(user_id=1060173cf2504bf58cdd3b34e525609f)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:44.092385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6a1116-39f1-4cf5-9180-b5077251da9a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:44.221284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1050/2102] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:44 2022] POST /identity/v3/projects => generated 369 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:44.222373 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea5e01b-2389-428f-917f-eabcc9d2c74e admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:44.233135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:44.287515 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea5e01b-2389-428f-917f-eabcc9d2c74e admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:44.292764 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b207d11b-8fbf-45b9-af2a-50b3b6242dfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:44.292956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b207d11b-8fbf-45b9-af2a-50b3b6242dfd admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:44.293198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b207d11b-8fbf-45b9-af2a-50b3b6242dfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:44.293460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b207d11b-8fbf-45b9-af2a-50b3b6242dfd admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:44.294071 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b207d11b-8fbf-45b9-af2a-50b3b6242dfd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:44.296003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b207d11b-8fbf-45b9-af2a-50b3b6242dfd admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:44.328002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9d6a1116-39f1-4cf5-9180-b5077251da9a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1060173cf2504bf58cdd3b34e525609f'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:44.338088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9d6a1116-39f1-4cf5-9180-b5077251da9a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1060173cf2504bf58cdd3b34e525609f'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:44.357644 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9d6a1116-39f1-4cf5-9180-b5077251da9a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1060173cf2504bf58cdd3b34e525609f'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:44.380592 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1051/2103] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:44 2022] POST /identity/v3/users => generated 285 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:44.384932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b207d11b-8fbf-45b9-af2a-50b3b6242dfd admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:44.395995 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b207d11b-8fbf-45b9-af2a-50b3b6242dfd admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:44.403942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32800ed-eb26-4a8c-8e25-b148cdaf19c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:44.409902 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32800ed-eb26-4a8c-8e25-b148cdaf19c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:44.409902 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32800ed-eb26-4a8c-8e25-b148cdaf19c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:44.409902 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32800ed-eb26-4a8c-8e25-b148cdaf19c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:44.409902 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32800ed-eb26-4a8c-8e25-b148cdaf19c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:44.411619 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32800ed-eb26-4a8c-8e25-b148cdaf19c6 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:44.423265 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1052/2104] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:44 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:44.427189 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32800ed-eb26-4a8c-8e25-b148cdaf19c6 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:44.436986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32800ed-eb26-4a8c-8e25-b148cdaf19c6 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:44.442321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c9ac76-9fc2-46d9-8284-e4533189c00b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:44.442999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c9ac76-9fc2-46d9-8284-e4533189c00b admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:44.443280 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c9ac76-9fc2-46d9-8284-e4533189c00b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:44.443577 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c9ac76-9fc2-46d9-8284-e4533189c00b admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:44.444260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c9ac76-9fc2-46d9-8284-e4533189c00b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:44.445842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c9ac76-9fc2-46d9-8284-e4533189c00b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:44.456710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1053/2105] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:44 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:44.461430 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c9ac76-9fc2-46d9-8284-e4533189c00b admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:44.461845 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9d6a1116-39f1-4cf5-9180-b5077251da9a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1060173cf2504bf58cdd3b34e525609f'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:44.462709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9d6a1116-39f1-4cf5-9180-b5077251da9a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1060173cf2504bf58cdd3b34e525609f'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:44.472986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c9ac76-9fc2-46d9-8284-e4533189c00b admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:44.479708 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1053/2106] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:44 2022] DELETE /identity/v3/users/1060173cf2504bf58cdd3b34e525609f => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:44.484066 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b358230-4bc4-4bdf-bd11-264c9183c3d8 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:44.484442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b358230-4bc4-4bdf-bd11-264c9183c3d8 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:44.484714 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b358230-4bc4-4bdf-bd11-264c9183c3d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:44.485876 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13a430c-6cec-422e-b24c-38e948708894 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:44.486549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13a430c-6cec-422e-b24c-38e948708894 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:44.487073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13a430c-6cec-422e-b24c-38e948708894 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:44.487415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13a430c-6cec-422e-b24c-38e948708894 admin admin] PATH_INFO: `/v3/projects/747410d56cdf42da95c8d1282d3d5d06/users/01ffe9b26c36459fafa185e0e51b2f65/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:44.504426 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13a430c-6cec-422e-b24c-38e948708894 admin admin] RBAC: Authorizing `identity:create_grant(project_id=747410d56cdf42da95c8d1282d3d5d06, user_id=01ffe9b26c36459fafa185e0e51b2f65, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:44.505823 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-8b358230-4bc4-4bdf-bd11-264c9183c3d8 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:44.507026 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-8b358230-4bc4-4bdf-bd11-264c9183c3d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:44.507148 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13a430c-6cec-422e-b24c-38e948708894 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:44.569497 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1054/2107] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:44.579868 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b358230-4bc4-4bdf-bd11-264c9183c3d8 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:44.589029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1054/2108] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:44 2022] PUT /identity/v3/projects/747410d56cdf42da95c8d1282d3d5d06/users/01ffe9b26c36459fafa185e0e51b2f65/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:44.589358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-8b358230-4bc4-4bdf-bd11-264c9183c3d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:44.597980 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97adca79-b6c7-4f3f-879f-1979128c94e2 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:44.598323 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97adca79-b6c7-4f3f-879f-1979128c94e2 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:44.598598 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97adca79-b6c7-4f3f-879f-1979128c94e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:44.626778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-97adca79-b6c7-4f3f-879f-1979128c94e2 None None] MFA Rules not processed for user `01ffe9b26c36459fafa185e0e51b2f65`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:44.627999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-97adca79-b6c7-4f3f-879f-1979128c94e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:44.654958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b358230-4bc4-4bdf-bd11-264c9183c3d8 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:44.668732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:44.704237 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:44.725353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1055/2109] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:44 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:44.734721 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac984afd-fa38-4821-884d-b75de8c3d16d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:44.735396 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac984afd-fa38-4821-884d-b75de8c3d16d service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:44.735739 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac984afd-fa38-4821-884d-b75de8c3d16d service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:44.736079 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac984afd-fa38-4821-884d-b75de8c3d16d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:44.738685 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97adca79-b6c7-4f3f-879f-1979128c94e2 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:44.740245 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-97adca79-b6c7-4f3f-879f-1979128c94e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:44.805632 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97adca79-b6c7-4f3f-879f-1979128c94e2 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:44.808433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac984afd-fa38-4821-884d-b75de8c3d16d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:44.810452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac984afd-fa38-4821-884d-b75de8c3d16d service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:44.811979 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1749487-4e8d-4285-884e-1ee249514b69 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6365b2328f34ab9b1b73ececbf5cc30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:44.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1749487-4e8d-4285-884e-1ee249514b69 service glance] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:44.813062 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1749487-4e8d-4285-884e-1ee249514b69 service glance] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:44.817841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1749487-4e8d-4285-884e-1ee249514b69 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:44.823869 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1749487-4e8d-4285-884e-1ee249514b69 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:44.824480 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1055/2110] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:26:44.825717 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1749487-4e8d-4285-884e-1ee249514b69 service glance] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:44.846079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1056/2111] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:44 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:26:45.310583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac984afd-fa38-4821-884d-b75de8c3d16d service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:45.312142 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ac984afd-fa38-4821-884d-b75de8c3d16d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:45.325427 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac984afd-fa38-4821-884d-b75de8c3d16d service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:45.330936 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435297ff-8f2f-4ff7-9eb8-5c198e51e737 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:45.331575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435297ff-8f2f-4ff7-9eb8-5c198e51e737 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:45.332064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435297ff-8f2f-4ff7-9eb8-5c198e51e737 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:45.332193 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435297ff-8f2f-4ff7-9eb8-5c198e51e737 admin admin] PATH_INFO: `/v3/projects/f712e114877b450182add63063b4780d` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:45.333262 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435297ff-8f2f-4ff7-9eb8-5c198e51e737 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f712e114877b450182add63063b4780d)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:45.335901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435297ff-8f2f-4ff7-9eb8-5c198e51e737 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:45.514563 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-435297ff-8f2f-4ff7-9eb8-5c198e51e737 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f712e114877b450182add63063b4780d', 'request_id': 'req-435297ff-8f2f-4ff7-9eb8-5c198e51e737'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:45.619506 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-435297ff-8f2f-4ff7-9eb8-5c198e51e737 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f712e114877b450182add63063b4780d', 'request_id': 'req-435297ff-8f2f-4ff7-9eb8-5c198e51e737'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:45.626551 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-435297ff-8f2f-4ff7-9eb8-5c198e51e737 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f712e114877b450182add63063b4780d', 'request_id': 'req-435297ff-8f2f-4ff7-9eb8-5c198e51e737'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:45.640881 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-435297ff-8f2f-4ff7-9eb8-5c198e51e737 admin admin] The token cache is being invalidate because project f712e114877b450182add63063b4780d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:45.642028 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-435297ff-8f2f-4ff7-9eb8-5c198e51e737 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:45.643521 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-435297ff-8f2f-4ff7-9eb8-5c198e51e737 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:45.645788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1056/2112] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:45 2022] DELETE /identity/v3/projects/f712e114877b450182add63063b4780d => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:45.649755 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1749487-4e8d-4285-884e-1ee249514b69 service glance] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:45.661417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1749487-4e8d-4285-884e-1ee249514b69 service glance] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:45.718902 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ece569d-87a6-4c12-adb6-e20f51f01f35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:45.719536 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ece569d-87a6-4c12-adb6-e20f51f01f35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:45.719810 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ece569d-87a6-4c12-adb6-e20f51f01f35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:45.720078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ece569d-87a6-4c12-adb6-e20f51f01f35 admin admin] PATH_INFO: `/v3/users/654edc3fba324776bb66e300e22ac5e8` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:45.721291 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ece569d-87a6-4c12-adb6-e20f51f01f35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=654edc3fba324776bb66e300e22ac5e8)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:45.723738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ece569d-87a6-4c12-adb6-e20f51f01f35 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:45.950151 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1ece569d-87a6-4c12-adb6-e20f51f01f35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '654edc3fba324776bb66e300e22ac5e8'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:45.960758 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1ece569d-87a6-4c12-adb6-e20f51f01f35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '654edc3fba324776bb66e300e22ac5e8'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:45.961527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1ece569d-87a6-4c12-adb6-e20f51f01f35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '654edc3fba324776bb66e300e22ac5e8'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:45.976177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1ece569d-87a6-4c12-adb6-e20f51f01f35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '654edc3fba324776bb66e300e22ac5e8'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:46.050037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1ece569d-87a6-4c12-adb6-e20f51f01f35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '654edc3fba324776bb66e300e22ac5e8'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:46.052448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:46.057720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1057/2113] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:45 2022] DELETE /identity/v3/users/654edc3fba324776bb66e300e22ac5e8 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:46.476288 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435297ff-8f2f-4ff7-9eb8-5c198e51e737 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:46.477876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-435297ff-8f2f-4ff7-9eb8-5c198e51e737 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:46.492293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435297ff-8f2f-4ff7-9eb8-5c198e51e737 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:46.554595 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35620ca8-8725-48fc-a229-cce6e057838b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:46.555309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35620ca8-8725-48fc-a229-cce6e057838b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:46.555509 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35620ca8-8725-48fc-a229-cce6e057838b admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:46.555780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35620ca8-8725-48fc-a229-cce6e057838b admin admin] PATH_INFO: `/v3/projects/02fb285c7092440181b91bb18f25e993` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:46.556985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35620ca8-8725-48fc-a229-cce6e057838b admin admin] RBAC: Authorizing `identity:delete_project(project_id=02fb285c7092440181b91bb18f25e993)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:46.558890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35620ca8-8725-48fc-a229-cce6e057838b admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:46.572191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:46.710832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-35620ca8-8725-48fc-a229-cce6e057838b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02fb285c7092440181b91bb18f25e993', 'request_id': 'req-35620ca8-8725-48fc-a229-cce6e057838b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:46.819500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-35620ca8-8725-48fc-a229-cce6e057838b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02fb285c7092440181b91bb18f25e993', 'request_id': 'req-35620ca8-8725-48fc-a229-cce6e057838b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:46.827142 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-35620ca8-8725-48fc-a229-cce6e057838b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02fb285c7092440181b91bb18f25e993', 'request_id': 'req-35620ca8-8725-48fc-a229-cce6e057838b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:46.846211 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-35620ca8-8725-48fc-a229-cce6e057838b admin admin] The token cache is being invalidate because project 02fb285c7092440181b91bb18f25e993 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:46.846807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-35620ca8-8725-48fc-a229-cce6e057838b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:46.848633 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-35620ca8-8725-48fc-a229-cce6e057838b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:46.850557 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1057/2114] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:46 2022] DELETE /identity/v3/projects/02fb285c7092440181b91bb18f25e993 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:46.857536 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2806ad6b-e304-4d83-bd16-4c7f447c6a32 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:46.857978 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2806ad6b-e304-4d83-bd16-4c7f447c6a32 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:46.858336 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2806ad6b-e304-4d83-bd16-4c7f447c6a32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:46.879942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-2806ad6b-e304-4d83-bd16-4c7f447c6a32 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:46.881177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-2806ad6b-e304-4d83-bd16-4c7f447c6a32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:46.962481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1058/2115] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:46.975532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de3e543-148c-4637-b5c5-3cf394d771a7 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:46.975809 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de3e543-148c-4637-b5c5-3cf394d771a7 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:46.976084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de3e543-148c-4637-b5c5-3cf394d771a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:46.997480 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-0de3e543-148c-4637-b5c5-3cf394d771a7 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:46.998692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-0de3e543-148c-4637-b5c5-3cf394d771a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:47.008469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1058/2116] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:47.014641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2806ad6b-e304-4d83-bd16-4c7f447c6a32 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:47.025347 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2806ad6b-e304-4d83-bd16-4c7f447c6a32 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:47.030281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e278bd-d464-4513-838e-9771fdbf92d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:47.030955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e278bd-d464-4513-838e-9771fdbf92d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:47.031234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e278bd-d464-4513-838e-9771fdbf92d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:47.031503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e278bd-d464-4513-838e-9771fdbf92d8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:47.032169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e278bd-d464-4513-838e-9771fdbf92d8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:26:47.032469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e278bd-d464-4513-838e-9771fdbf92d8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:47.034179 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e278bd-d464-4513-838e-9771fdbf92d8 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:47.051104 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1059/2117] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:26:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:26:47.052465 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de3e543-148c-4637-b5c5-3cf394d771a7 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:47.062180 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de3e543-148c-4637-b5c5-3cf394d771a7 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:47.067315 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5303e43-3683-4241-84ca-c6bfce095a9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:47.068147 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5303e43-3683-4241-84ca-c6bfce095a9a admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:47.068429 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5303e43-3683-4241-84ca-c6bfce095a9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:47.068777 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5303e43-3683-4241-84ca-c6bfce095a9a admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:47.069413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5303e43-3683-4241-84ca-c6bfce095a9a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:47.071362 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5303e43-3683-4241-84ca-c6bfce095a9a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:47.223479 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1059/2118] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:47 2022] POST /identity/v3/projects => generated 373 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:47.224302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e278bd-d464-4513-838e-9771fdbf92d8 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:47.238689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e278bd-d464-4513-838e-9771fdbf92d8 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:47.311332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d755115c-6219-4430-841a-4eec6f89c363 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:47.311998 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d755115c-6219-4430-841a-4eec6f89c363 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:47.312362 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d755115c-6219-4430-841a-4eec6f89c363 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:47.312831 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d755115c-6219-4430-841a-4eec6f89c363 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:47.313930 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d755115c-6219-4430-841a-4eec6f89c363 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:47.317297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d755115c-6219-4430-841a-4eec6f89c363 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:47.397335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1060/2119] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:47 2022] POST /identity/v3/users => generated 287 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:47.398575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5303e43-3683-4241-84ca-c6bfce095a9a admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:47.409482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5303e43-3683-4241-84ca-c6bfce095a9a admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:47.414590 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d77c696-b62f-4a40-8aa7-ab785d2f9dad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:47.415187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d77c696-b62f-4a40-8aa7-ab785d2f9dad admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:47.415465 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d77c696-b62f-4a40-8aa7-ab785d2f9dad admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:47.415819 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d77c696-b62f-4a40-8aa7-ab785d2f9dad admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:47.416539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d77c696-b62f-4a40-8aa7-ab785d2f9dad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:47.418138 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d77c696-b62f-4a40-8aa7-ab785d2f9dad admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:47.429271 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1060/2120] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:47 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:47.433618 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d755115c-6219-4430-841a-4eec6f89c363 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:47.443934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d755115c-6219-4430-841a-4eec6f89c363 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:47.448631 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38ef3b8-bb62-481d-b789-957290f18054 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:47.449228 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38ef3b8-bb62-481d-b789-957290f18054 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:47.449499 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38ef3b8-bb62-481d-b789-957290f18054 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:47.449765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38ef3b8-bb62-481d-b789-957290f18054 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:47.450394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38ef3b8-bb62-481d-b789-957290f18054 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:47.452009 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38ef3b8-bb62-481d-b789-957290f18054 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:47.462893 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1061/2121] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:47.467185 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d77c696-b62f-4a40-8aa7-ab785d2f9dad admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:47.477255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d77c696-b62f-4a40-8aa7-ab785d2f9dad admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:47.481928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e38bb0-78dc-4f59-a010-25b1402c83f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:47.482578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e38bb0-78dc-4f59-a010-25b1402c83f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:47.482851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e38bb0-78dc-4f59-a010-25b1402c83f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:47.483120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e38bb0-78dc-4f59-a010-25b1402c83f5 admin admin] PATH_INFO: `/v3/projects/9494d14579844a7e94ce753bc9e0269b/users/b9bba41544f84dc195dd3c8bf75d2525/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:47.498142 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e38bb0-78dc-4f59-a010-25b1402c83f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9494d14579844a7e94ce753bc9e0269b, user_id=b9bba41544f84dc195dd3c8bf75d2525, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:47.500800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e38bb0-78dc-4f59-a010-25b1402c83f5 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:47.588538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1061/2122] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:47 2022] PUT /identity/v3/projects/9494d14579844a7e94ce753bc9e0269b/users/b9bba41544f84dc195dd3c8bf75d2525/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:47.598804 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e52f14-81e4-49ff-bcb2-0cbec5bf3631 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:47.599079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e52f14-81e4-49ff-bcb2-0cbec5bf3631 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:47.599343 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e52f14-81e4-49ff-bcb2-0cbec5bf3631 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:47.627445 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-74e52f14-81e4-49ff-bcb2-0cbec5bf3631 None None] MFA Rules not processed for user `b9bba41544f84dc195dd3c8bf75d2525`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:47.628572 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-74e52f14-81e4-49ff-bcb2-0cbec5bf3631 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:47.686459 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:47.697507 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1062/2123] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:47 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:47.702401 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e38bb0-78dc-4f59-a010-25b1402c83f5 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:47.767597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e38bb0-78dc-4f59-a010-25b1402c83f5 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:47.772594 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b042ccbc-f694-461a-b5ee-9a6189439eeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:47.773245 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b042ccbc-f694-461a-b5ee-9a6189439eeb admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:47.773528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b042ccbc-f694-461a-b5ee-9a6189439eeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:47.773827 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b042ccbc-f694-461a-b5ee-9a6189439eeb admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:47.774466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b042ccbc-f694-461a-b5ee-9a6189439eeb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:47.776571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b042ccbc-f694-461a-b5ee-9a6189439eeb admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:47.870091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1062/2124] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:47 2022] POST /identity/v3/projects => generated 373 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:47.875640 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e52f14-81e4-49ff-bcb2-0cbec5bf3631 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:47.887193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e52f14-81e4-49ff-bcb2-0cbec5bf3631 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:47.949259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec099f88-8cad-44a4-abad-f019367703e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:47.949869 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec099f88-8cad-44a4-abad-f019367703e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:47.950144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec099f88-8cad-44a4-abad-f019367703e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:47.950407 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec099f88-8cad-44a4-abad-f019367703e8 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:47.951035 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec099f88-8cad-44a4-abad-f019367703e8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:47.953013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec099f88-8cad-44a4-abad-f019367703e8 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:48.057089 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1063/2125] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:47 2022] POST /identity/v3/users => generated 293 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:48.060910 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b042ccbc-f694-461a-b5ee-9a6189439eeb admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:48.072760 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b042ccbc-f694-461a-b5ee-9a6189439eeb admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:48.078200 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d901a6f-c5cb-4ac8-b799-f60b11fc9da7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:48.078815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d901a6f-c5cb-4ac8-b799-f60b11fc9da7 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:48.079092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d901a6f-c5cb-4ac8-b799-f60b11fc9da7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:48.079364 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d901a6f-c5cb-4ac8-b799-f60b11fc9da7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:48.080014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d901a6f-c5cb-4ac8-b799-f60b11fc9da7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:48.081660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d901a6f-c5cb-4ac8-b799-f60b11fc9da7 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:48.095442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1063/2126] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:48 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:48.100207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec099f88-8cad-44a4-abad-f019367703e8 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:48.110351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec099f88-8cad-44a4-abad-f019367703e8 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:48.115665 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae536ca-9b74-44e6-8c8a-551dc2010e9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:48.116344 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae536ca-9b74-44e6-8c8a-551dc2010e9c admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:48.116627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae536ca-9b74-44e6-8c8a-551dc2010e9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:48.116902 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae536ca-9b74-44e6-8c8a-551dc2010e9c admin admin] PATH_INFO: `/v3/projects/1e644c5ce24b4aae9dabf2d617b8e1c3/users/c80113930f274aabb5d7dd1225ec632f/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:48.131417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae536ca-9b74-44e6-8c8a-551dc2010e9c admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e644c5ce24b4aae9dabf2d617b8e1c3, user_id=c80113930f274aabb5d7dd1225ec632f, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:48.134092 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae536ca-9b74-44e6-8c8a-551dc2010e9c admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:48.205345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1064/2127] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:48 2022] PUT /identity/v3/projects/1e644c5ce24b4aae9dabf2d617b8e1c3/users/c80113930f274aabb5d7dd1225ec632f/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:48.209304 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d901a6f-c5cb-4ac8-b799-f60b11fc9da7 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:48.219318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d901a6f-c5cb-4ac8-b799-f60b11fc9da7 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:48.227796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:48.278371 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be34a986-e22f-4938-a887-94c605711289 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:48.278976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be34a986-e22f-4938-a887-94c605711289 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:48.279249 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be34a986-e22f-4938-a887-94c605711289 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:48.279533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be34a986-e22f-4938-a887-94c605711289 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:48.280185 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be34a986-e22f-4938-a887-94c605711289 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:48.281761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be34a986-e22f-4938-a887-94c605711289 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:48.293218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1064/2128] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:48 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:26:48.297442 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae536ca-9b74-44e6-8c8a-551dc2010e9c admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:48.311912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae536ca-9b74-44e6-8c8a-551dc2010e9c admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:48.317338 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16010567-87c0-4525-86a2-782a243cf04b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:48.318101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16010567-87c0-4525-86a2-782a243cf04b admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:48.318559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16010567-87c0-4525-86a2-782a243cf04b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:48.321678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16010567-87c0-4525-86a2-782a243cf04b admin admin] PATH_INFO: `/v3/projects/1e644c5ce24b4aae9dabf2d617b8e1c3/users/c80113930f274aabb5d7dd1225ec632f/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:48.321678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16010567-87c0-4525-86a2-782a243cf04b admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e644c5ce24b4aae9dabf2d617b8e1c3, user_id=c80113930f274aabb5d7dd1225ec632f, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:48.324366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16010567-87c0-4525-86a2-782a243cf04b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:48.353175 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1065/2129] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:48 2022] PUT /identity/v3/projects/1e644c5ce24b4aae9dabf2d617b8e1c3/users/c80113930f274aabb5d7dd1225ec632f/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:48.359517 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee9fcff-334b-426a-a8de-7c96fd0649e6 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:48.359802 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee9fcff-334b-426a-a8de-7c96fd0649e6 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:48.360072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee9fcff-334b-426a-a8de-7c96fd0649e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:48.391980 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-7ee9fcff-334b-426a-a8de-7c96fd0649e6 None None] MFA Rules not processed for user `c80113930f274aabb5d7dd1225ec632f`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:48.393251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-7ee9fcff-334b-426a-a8de-7c96fd0649e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:48.487476 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1065/2130] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:48 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:48.499273 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16010567-87c0-4525-86a2-782a243cf04b admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:48.501472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-16010567-87c0-4525-86a2-782a243cf04b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:48.566904 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16010567-87c0-4525-86a2-782a243cf04b admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:48.570915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6868a72d-0909-47b8-9100-e2304f41c9d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:48.571537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6868a72d-0909-47b8-9100-e2304f41c9d1 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:48.571816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6868a72d-0909-47b8-9100-e2304f41c9d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:48.572086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6868a72d-0909-47b8-9100-e2304f41c9d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:48.633441 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6868a72d-0909-47b8-9100-e2304f41c9d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:48.634903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6868a72d-0909-47b8-9100-e2304f41c9d1 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:48.639699 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:48.644454 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1066/2131] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:48 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:51.479572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee9fcff-334b-426a-a8de-7c96fd0649e6 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:51.481117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-7ee9fcff-334b-426a-a8de-7c96fd0649e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:51.550181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee9fcff-334b-426a-a8de-7c96fd0649e6 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:51.554741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7295f0ee-c188-4a5d-8a1c-d9510be91fce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:51.555371 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7295f0ee-c188-4a5d-8a1c-d9510be91fce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:51.555641 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7295f0ee-c188-4a5d-8a1c-d9510be91fce admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:51.555908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7295f0ee-c188-4a5d-8a1c-d9510be91fce admin admin] PATH_INFO: `/v3/users/04456917f3c24a5dbb50e3a3eb187b10` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:51.557165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7295f0ee-c188-4a5d-8a1c-d9510be91fce admin admin] RBAC: Authorizing `identity:delete_user(user_id=04456917f3c24a5dbb50e3a3eb187b10)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:51.559084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7295f0ee-c188-4a5d-8a1c-d9510be91fce admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:51.714446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:51.723704 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7295f0ee-c188-4a5d-8a1c-d9510be91fce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '04456917f3c24a5dbb50e3a3eb187b10'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:51.731651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7295f0ee-c188-4a5d-8a1c-d9510be91fce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '04456917f3c24a5dbb50e3a3eb187b10'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:51.740342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6868a72d-0909-47b8-9100-e2304f41c9d1 service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:51.744294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7295f0ee-c188-4a5d-8a1c-d9510be91fce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '04456917f3c24a5dbb50e3a3eb187b10'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:51.751266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6868a72d-0909-47b8-9100-e2304f41c9d1 service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:51.755949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42184946-3968-482e-b7ce-6d979a316d50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:51.756749 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42184946-3968-482e-b7ce-6d979a316d50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:51.757057 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42184946-3968-482e-b7ce-6d979a316d50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:51.757325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42184946-3968-482e-b7ce-6d979a316d50 admin admin] PATH_INFO: `/v3/users/b9bba41544f84dc195dd3c8bf75d2525` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:51.758972 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42184946-3968-482e-b7ce-6d979a316d50 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b9bba41544f84dc195dd3c8bf75d2525)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:51.761088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42184946-3968-482e-b7ce-6d979a316d50 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:51.802886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7295f0ee-c188-4a5d-8a1c-d9510be91fce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '04456917f3c24a5dbb50e3a3eb187b10'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:51.803542 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7295f0ee-c188-4a5d-8a1c-d9510be91fce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '04456917f3c24a5dbb50e3a3eb187b10'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:51.809921 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1066/2132] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:51 2022] DELETE /identity/v3/users/04456917f3c24a5dbb50e3a3eb187b10 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:51.814090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c7ccc9-88eb-4482-a184-0233b2b1b30c None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:51.814370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c7ccc9-88eb-4482-a184-0233b2b1b30c None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:51.814699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c7ccc9-88eb-4482-a184-0233b2b1b30c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:51.834275 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-56c7ccc9-88eb-4482-a184-0233b2b1b30c None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:51.835471 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-56c7ccc9-88eb-4482-a184-0233b2b1b30c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:51.893570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1067/2133] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:51.903879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c7ccc9-88eb-4482-a184-0233b2b1b30c None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:51.905275 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-56c7ccc9-88eb-4482-a184-0233b2b1b30c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:51.965077 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c7ccc9-88eb-4482-a184-0233b2b1b30c None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:51.969617 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d56d64-cff1-4d30-b139-ba6dc1104636 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:51.970250 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d56d64-cff1-4d30-b139-ba6dc1104636 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:51.970525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d56d64-cff1-4d30-b139-ba6dc1104636 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:51.970796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d56d64-cff1-4d30-b139-ba6dc1104636 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:51.980707 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d56d64-cff1-4d30-b139-ba6dc1104636 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:51.982296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d56d64-cff1-4d30-b139-ba6dc1104636 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:51.992282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1068/2134] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:26:52.029874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-42184946-3968-482e-b7ce-6d979a316d50 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b9bba41544f84dc195dd3c8bf75d2525'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:52.039842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-42184946-3968-482e-b7ce-6d979a316d50 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b9bba41544f84dc195dd3c8bf75d2525'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:52.040702 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-42184946-3968-482e-b7ce-6d979a316d50 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b9bba41544f84dc195dd3c8bf75d2525'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:52.057931 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-42184946-3968-482e-b7ce-6d979a316d50 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b9bba41544f84dc195dd3c8bf75d2525'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:52.137779 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-42184946-3968-482e-b7ce-6d979a316d50 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b9bba41544f84dc195dd3c8bf75d2525'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:52.144186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1067/2135] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:51 2022] DELETE /identity/v3/users/b9bba41544f84dc195dd3c8bf75d2525 => generated 0 bytes in 404 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:52.148855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5e33b8-5b73-477a-a116-bb46ee631d4c None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:52.149170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5e33b8-5b73-477a-a116-bb46ee631d4c None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:52.149499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5e33b8-5b73-477a-a116-bb46ee631d4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:52.168250 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-ad5e33b8-5b73-477a-a116-bb46ee631d4c None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:52.169445 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ad5e33b8-5b73-477a-a116-bb46ee631d4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:52.224384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:52.237911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1069/2136] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:52.249295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42184946-3968-482e-b7ce-6d979a316d50 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:52.252503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-42184946-3968-482e-b7ce-6d979a316d50 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:52.327112 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42184946-3968-482e-b7ce-6d979a316d50 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:52.330882 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6f6be1-dc8c-4053-93fb-98f24b2c55db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:52.331487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6f6be1-dc8c-4053-93fb-98f24b2c55db service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:52.331763 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6f6be1-dc8c-4053-93fb-98f24b2c55db service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:52.332029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6f6be1-dc8c-4053-93fb-98f24b2c55db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:52.342312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6f6be1-dc8c-4053-93fb-98f24b2c55db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:52.343861 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6f6be1-dc8c-4053-93fb-98f24b2c55db service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:52.355688 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1068/2137] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:26:52.517735 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5e33b8-5b73-477a-a116-bb46ee631d4c None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:52.519317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ad5e33b8-5b73-477a-a116-bb46ee631d4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:52.531351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5e33b8-5b73-477a-a116-bb46ee631d4c None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:52.536095 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a8d11b-7da9-4f88-b00b-7523401288d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:52.537124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a8d11b-7da9-4f88-b00b-7523401288d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:52.537267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a8d11b-7da9-4f88-b00b-7523401288d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:52.537599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a8d11b-7da9-4f88-b00b-7523401288d8 admin admin] PATH_INFO: `/v3/projects/8ddaa0706bd54806973c06cb1dca3ed6` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:52.538801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a8d11b-7da9-4f88-b00b-7523401288d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ddaa0706bd54806973c06cb1dca3ed6)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:52.541313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a8d11b-7da9-4f88-b00b-7523401288d8 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:52.674346 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-73a8d11b-7da9-4f88-b00b-7523401288d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ddaa0706bd54806973c06cb1dca3ed6', 'request_id': 'req-73a8d11b-7da9-4f88-b00b-7523401288d8'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:52.743955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-73a8d11b-7da9-4f88-b00b-7523401288d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ddaa0706bd54806973c06cb1dca3ed6', 'request_id': 'req-73a8d11b-7da9-4f88-b00b-7523401288d8'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:52.750335 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-73a8d11b-7da9-4f88-b00b-7523401288d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ddaa0706bd54806973c06cb1dca3ed6', 'request_id': 'req-73a8d11b-7da9-4f88-b00b-7523401288d8'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:52.762671 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-73a8d11b-7da9-4f88-b00b-7523401288d8 admin admin] The token cache is being invalidate because project 8ddaa0706bd54806973c06cb1dca3ed6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:52.763003 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-73a8d11b-7da9-4f88-b00b-7523401288d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:52.763696 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-73a8d11b-7da9-4f88-b00b-7523401288d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:52.765470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1070/2138] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:52 2022] DELETE /identity/v3/projects/8ddaa0706bd54806973c06cb1dca3ed6 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:52.770317 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7528ab40-a98e-479a-a79f-041bea903ef4 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:52.770790 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7528ab40-a98e-479a-a79f-041bea903ef4 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:52.776866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7528ab40-a98e-479a-a79f-041bea903ef4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:52.791725 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-7528ab40-a98e-479a-a79f-041bea903ef4 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:52.792954 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7528ab40-a98e-479a-a79f-041bea903ef4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:52.853666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1069/2139] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:52.861437 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6643fbf1-7536-4b08-90f7-63af019dc47a None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:52.861716 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6643fbf1-7536-4b08-90f7-63af019dc47a None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:52.861986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6643fbf1-7536-4b08-90f7-63af019dc47a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:52.882613 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-6643fbf1-7536-4b08-90f7-63af019dc47a None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:52.883819 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6643fbf1-7536-4b08-90f7-63af019dc47a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:52.891694 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1071/2140] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:52.895935 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7528ab40-a98e-479a-a79f-041bea903ef4 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:52.906332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7528ab40-a98e-479a-a79f-041bea903ef4 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:52.910952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fa143f-61e6-4aa8-bd02-f82ff8575fd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:52.911977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fa143f-61e6-4aa8-bd02-f82ff8575fd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:52.912125 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fa143f-61e6-4aa8-bd02-f82ff8575fd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:52.912431 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fa143f-61e6-4aa8-bd02-f82ff8575fd6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:52.913062 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fa143f-61e6-4aa8-bd02-f82ff8575fd6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:26:52.913361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fa143f-61e6-4aa8-bd02-f82ff8575fd6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:52.915590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fa143f-61e6-4aa8-bd02-f82ff8575fd6 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:52.927215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1070/2141] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:26:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:26:52.931627 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6643fbf1-7536-4b08-90f7-63af019dc47a None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:52.939805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fa143f-61e6-4aa8-bd02-f82ff8575fd6 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:52.941645 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-56fa143f-61e6-4aa8-bd02-f82ff8575fd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:52.941810 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6643fbf1-7536-4b08-90f7-63af019dc47a None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:52.946658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8505ec-9cd7-4ebc-82cb-7e1fdf8aa13f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:52.947379 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8505ec-9cd7-4ebc-82cb-7e1fdf8aa13f admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:52.947659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8505ec-9cd7-4ebc-82cb-7e1fdf8aa13f admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:52.947928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8505ec-9cd7-4ebc-82cb-7e1fdf8aa13f admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:52.948588 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8505ec-9cd7-4ebc-82cb-7e1fdf8aa13f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:52.950490 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8505ec-9cd7-4ebc-82cb-7e1fdf8aa13f admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:52.951188 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:52.957656 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fa143f-61e6-4aa8-bd02-f82ff8575fd6 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:52.963445 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a9b352-537b-4f20-a793-ee50a564e2a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:52.964087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a9b352-537b-4f20-a793-ee50a564e2a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:52.964483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a9b352-537b-4f20-a793-ee50a564e2a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:52.964959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a9b352-537b-4f20-a793-ee50a564e2a0 admin admin] PATH_INFO: `/v3/projects/9494d14579844a7e94ce753bc9e0269b` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:52.966327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a9b352-537b-4f20-a793-ee50a564e2a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9494d14579844a7e94ce753bc9e0269b)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:52.969095 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a9b352-537b-4f20-a793-ee50a564e2a0 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:53.033125 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1072/2142] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:52 2022] POST /identity/v3/projects => generated 391 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:53.035859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8505ec-9cd7-4ebc-82cb-7e1fdf8aa13f admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:53.097156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8505ec-9cd7-4ebc-82cb-7e1fdf8aa13f admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:53.105307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-90a9b352-537b-4f20-a793-ee50a564e2a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9494d14579844a7e94ce753bc9e0269b', 'request_id': 'req-90a9b352-537b-4f20-a793-ee50a564e2a0'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:53.111025 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-90a9b352-537b-4f20-a793-ee50a564e2a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9494d14579844a7e94ce753bc9e0269b', 'request_id': 'req-90a9b352-537b-4f20-a793-ee50a564e2a0'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:53.118012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-90a9b352-537b-4f20-a793-ee50a564e2a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9494d14579844a7e94ce753bc9e0269b', 'request_id': 'req-90a9b352-537b-4f20-a793-ee50a564e2a0'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:53.153222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:53.159305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017ea71d-07da-44d6-9d52-0ad78996fdb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:53.159937 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017ea71d-07da-44d6-9d52-0ad78996fdb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:53.160244 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017ea71d-07da-44d6-9d52-0ad78996fdb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:53.160515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017ea71d-07da-44d6-9d52-0ad78996fdb0 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:53.161142 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017ea71d-07da-44d6-9d52-0ad78996fdb0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:53.163290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017ea71d-07da-44d6-9d52-0ad78996fdb0 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:53.207107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-90a9b352-537b-4f20-a793-ee50a564e2a0 admin admin] The token cache is being invalidate because project 9494d14579844a7e94ce753bc9e0269b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:53.207527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-90a9b352-537b-4f20-a793-ee50a564e2a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:53.208299 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-90a9b352-537b-4f20-a793-ee50a564e2a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:53.210251 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1071/2143] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:52 2022] DELETE /identity/v3/projects/9494d14579844a7e94ce753bc9e0269b => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:53.214438 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a9b352-537b-4f20-a793-ee50a564e2a0 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:53.215798 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-90a9b352-537b-4f20-a793-ee50a564e2a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:53.229857 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a9b352-537b-4f20-a793-ee50a564e2a0 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:53.234649 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d1c77c-d523-48ac-8087-7675a5ace27b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:53.235279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d1c77c-d523-48ac-8087-7675a5ace27b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:53.235556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d1c77c-d523-48ac-8087-7675a5ace27b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:53.236491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d1c77c-d523-48ac-8087-7675a5ace27b admin admin] PATH_INFO: `/v3/users/c80113930f274aabb5d7dd1225ec632f` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:53.237063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d1c77c-d523-48ac-8087-7675a5ace27b admin admin] RBAC: Authorizing `identity:delete_user(user_id=c80113930f274aabb5d7dd1225ec632f)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:53.239115 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d1c77c-d523-48ac-8087-7675a5ace27b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:53.258954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1073/2144] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:53 2022] POST /identity/v3/users => generated 296 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:53.263562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017ea71d-07da-44d6-9d52-0ad78996fdb0 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:53.270463 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-017ea71d-07da-44d6-9d52-0ad78996fdb0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:53.283215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017ea71d-07da-44d6-9d52-0ad78996fdb0 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:53.293918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650d4592-342d-4549-b3e0-b76d823b5325 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:53.294518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650d4592-342d-4549-b3e0-b76d823b5325 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:53.294799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650d4592-342d-4549-b3e0-b76d823b5325 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:53.295206 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650d4592-342d-4549-b3e0-b76d823b5325 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:53.295994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650d4592-342d-4549-b3e0-b76d823b5325 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:53.297905 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650d4592-342d-4549-b3e0-b76d823b5325 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:53.309421 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1074/2145] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:53 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:53.313691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650d4592-342d-4549-b3e0-b76d823b5325 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:53.324552 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650d4592-342d-4549-b3e0-b76d823b5325 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:53.329259 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca559c03-ff72-4e6a-8623-658bc34d91f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:53.329854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca559c03-ff72-4e6a-8623-658bc34d91f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:53.330130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca559c03-ff72-4e6a-8623-658bc34d91f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:53.330399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca559c03-ff72-4e6a-8623-658bc34d91f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:53.331024 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca559c03-ff72-4e6a-8623-658bc34d91f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:53.332638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca559c03-ff72-4e6a-8623-658bc34d91f7 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:53.343722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1075/2146] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:53 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:53.347708 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca559c03-ff72-4e6a-8623-658bc34d91f7 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:53.361239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca559c03-ff72-4e6a-8623-658bc34d91f7 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:53.365875 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467e4400-fa65-4b6f-9fce-143d48e5db44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:53.366516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467e4400-fa65-4b6f-9fce-143d48e5db44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:53.366792 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467e4400-fa65-4b6f-9fce-143d48e5db44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:53.367065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467e4400-fa65-4b6f-9fce-143d48e5db44 admin admin] PATH_INFO: `/v3/projects/0f6936a00a984984b27977fb51ab1eda/users/45c901989a7d41ceb95c68b99fa515e0/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:53.382663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467e4400-fa65-4b6f-9fce-143d48e5db44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f6936a00a984984b27977fb51ab1eda, user_id=45c901989a7d41ceb95c68b99fa515e0, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:53.385973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467e4400-fa65-4b6f-9fce-143d48e5db44 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:53.469238 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-d9d1c77c-d523-48ac-8087-7675a5ace27b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c80113930f274aabb5d7dd1225ec632f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:53.476496 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-d9d1c77c-d523-48ac-8087-7675a5ace27b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c80113930f274aabb5d7dd1225ec632f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:53.477212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-d9d1c77c-d523-48ac-8087-7675a5ace27b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c80113930f274aabb5d7dd1225ec632f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:53.484085 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1076/2147] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:53 2022] PUT /identity/v3/projects/0f6936a00a984984b27977fb51ab1eda/users/45c901989a7d41ceb95c68b99fa515e0/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:53.490659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-d9d1c77c-d523-48ac-8087-7675a5ace27b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c80113930f274aabb5d7dd1225ec632f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:53.496189 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a73659-3be6-47df-ab70-5dab396280be None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:53.496474 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a73659-3be6-47df-ab70-5dab396280be None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:53.496750 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a73659-3be6-47df-ab70-5dab396280be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:53.524822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-43a73659-3be6-47df-ab70-5dab396280be None None] MFA Rules not processed for user `45c901989a7d41ceb95c68b99fa515e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:53.526023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-43a73659-3be6-47df-ab70-5dab396280be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:53.563419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-d9d1c77c-d523-48ac-8087-7675a5ace27b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c80113930f274aabb5d7dd1225ec632f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:53.569980 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1072/2148] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:26:53 2022] DELETE /identity/v3/users/c80113930f274aabb5d7dd1225ec632f => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:26:53.598605 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1077/2149] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:53 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:26:53.612262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d1c77c-d523-48ac-8087-7675a5ace27b admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:53.613892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d9d1c77c-d523-48ac-8087-7675a5ace27b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:53.674812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d1c77c-d523-48ac-8087-7675a5ace27b admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:53.678382 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc90b194-f6be-4e23-8da5-1418e6983997 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:53.679086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc90b194-f6be-4e23-8da5-1418e6983997 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:53.679357 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc90b194-f6be-4e23-8da5-1418e6983997 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:53.679622 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc90b194-f6be-4e23-8da5-1418e6983997 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:53.689453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:53.735067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc90b194-f6be-4e23-8da5-1418e6983997 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:53.736532 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc90b194-f6be-4e23-8da5-1418e6983997 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:53.745866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1073/2150] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:53 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:26:54.085236 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a73659-3be6-47df-ab70-5dab396280be None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:54.086771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-43a73659-3be6-47df-ab70-5dab396280be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:54.158489 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a73659-3be6-47df-ab70-5dab396280be None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:54.193225 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:54.215803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd6f978-afcd-464d-887d-a2e9c9e46189 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:54.216457 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd6f978-afcd-464d-887d-a2e9c9e46189 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:54.216733 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd6f978-afcd-464d-887d-a2e9c9e46189 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:54.220855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd6f978-afcd-464d-887d-a2e9c9e46189 admin admin] PATH_INFO: `/v3/projects/1e644c5ce24b4aae9dabf2d617b8e1c3` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:54.220855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd6f978-afcd-464d-887d-a2e9c9e46189 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1e644c5ce24b4aae9dabf2d617b8e1c3)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:54.220855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd6f978-afcd-464d-887d-a2e9c9e46189 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:54.371795 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2dd6f978-afcd-464d-887d-a2e9c9e46189 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1e644c5ce24b4aae9dabf2d617b8e1c3', 'request_id': 'req-2dd6f978-afcd-464d-887d-a2e9c9e46189'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:54.464112 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2dd6f978-afcd-464d-887d-a2e9c9e46189 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1e644c5ce24b4aae9dabf2d617b8e1c3', 'request_id': 'req-2dd6f978-afcd-464d-887d-a2e9c9e46189'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:54.470410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2dd6f978-afcd-464d-887d-a2e9c9e46189 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1e644c5ce24b4aae9dabf2d617b8e1c3', 'request_id': 'req-2dd6f978-afcd-464d-887d-a2e9c9e46189'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:54.484906 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2dd6f978-afcd-464d-887d-a2e9c9e46189 admin admin] The token cache is being invalidate because project 1e644c5ce24b4aae9dabf2d617b8e1c3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:26:54.484906 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2dd6f978-afcd-464d-887d-a2e9c9e46189 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:54.485459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2dd6f978-afcd-464d-887d-a2e9c9e46189 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:26:54.489651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1078/2151] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:26:54 2022] DELETE /identity/v3/projects/1e644c5ce24b4aae9dabf2d617b8e1c3 => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:54.494706 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b7dc98-2fd7-42a8-a59f-f979bfefc618 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:54.495177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b7dc98-2fd7-42a8-a59f-f979bfefc618 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:54.495256 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b7dc98-2fd7-42a8-a59f-f979bfefc618 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:54.519636 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-69b7dc98-2fd7-42a8-a59f-f979bfefc618 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:54.521135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-69b7dc98-2fd7-42a8-a59f-f979bfefc618 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:54.584886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1074/2152] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:54.593488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f57c2ea-5886-4da9-ab65-0b998148ce44 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:54.593976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f57c2ea-5886-4da9-ab65-0b998148ce44 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:54.593976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f57c2ea-5886-4da9-ab65-0b998148ce44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:54.615458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-1f57c2ea-5886-4da9-ab65-0b998148ce44 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:54.616751 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1f57c2ea-5886-4da9-ab65-0b998148ce44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:54.625161 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1079/2153] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:54.632684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b7dc98-2fd7-42a8-a59f-f979bfefc618 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:54.642546 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b7dc98-2fd7-42a8-a59f-f979bfefc618 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:54.647094 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2742ba-87ef-4a14-ae13-38755a8fca23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:54.647932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2742ba-87ef-4a14-ae13-38755a8fca23 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:54.648344 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2742ba-87ef-4a14-ae13-38755a8fca23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:54.648616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2742ba-87ef-4a14-ae13-38755a8fca23 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:54.649807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2742ba-87ef-4a14-ae13-38755a8fca23 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:26:54.650101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2742ba-87ef-4a14-ae13-38755a8fca23 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:54.651703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2742ba-87ef-4a14-ae13-38755a8fca23 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:54.663906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1075/2154] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:26:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:26:54.671439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f57c2ea-5886-4da9-ab65-0b998148ce44 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:54.681213 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f57c2ea-5886-4da9-ab65-0b998148ce44 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:54.686075 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec859cb-1b73-4515-a7a3-abb5ca217e95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:54.686687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec859cb-1b73-4515-a7a3-abb5ca217e95 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:54.686965 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec859cb-1b73-4515-a7a3-abb5ca217e95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:54.687238 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec859cb-1b73-4515-a7a3-abb5ca217e95 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:54.687908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec859cb-1b73-4515-a7a3-abb5ca217e95 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:54.689866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec859cb-1b73-4515-a7a3-abb5ca217e95 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:54.802516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1080/2155] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:26:54 2022] POST /identity/v3/projects => generated 383 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:26:54.806941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2742ba-87ef-4a14-ae13-38755a8fca23 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:54.817482 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2742ba-87ef-4a14-ae13-38755a8fca23 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:54.880846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04854d7b-563b-4a21-b887-31c683fe08d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:54.881364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04854d7b-563b-4a21-b887-31c683fe08d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:54.881641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04854d7b-563b-4a21-b887-31c683fe08d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:54.881910 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04854d7b-563b-4a21-b887-31c683fe08d9 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:54.882543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04854d7b-563b-4a21-b887-31c683fe08d9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:54.884480 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04854d7b-563b-4a21-b887-31c683fe08d9 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:54.956601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1076/2156] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:26:54 2022] POST /identity/v3/users => generated 292 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:26:54.960760 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec859cb-1b73-4515-a7a3-abb5ca217e95 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:54.971836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec859cb-1b73-4515-a7a3-abb5ca217e95 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:54.976624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080ae79b-7de9-4a0a-910d-6f3be6bf5202 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:54.977229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080ae79b-7de9-4a0a-910d-6f3be6bf5202 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:54.977509 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080ae79b-7de9-4a0a-910d-6f3be6bf5202 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:54.977782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080ae79b-7de9-4a0a-910d-6f3be6bf5202 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:54.978447 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080ae79b-7de9-4a0a-910d-6f3be6bf5202 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:54.980195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080ae79b-7de9-4a0a-910d-6f3be6bf5202 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:54.992963 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1081/2157] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:54 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:54.997317 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04854d7b-563b-4a21-b887-31c683fe08d9 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:55.007131 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04854d7b-563b-4a21-b887-31c683fe08d9 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:55.012653 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3031d291-d3ef-469b-bfb0-81e249b7b06f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:55.013238 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3031d291-d3ef-469b-bfb0-81e249b7b06f admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:55.013511 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3031d291-d3ef-469b-bfb0-81e249b7b06f admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:55.013777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3031d291-d3ef-469b-bfb0-81e249b7b06f admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:55.014413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3031d291-d3ef-469b-bfb0-81e249b7b06f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:55.015973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3031d291-d3ef-469b-bfb0-81e249b7b06f admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:55.027172 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1077/2158] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:26:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:26:55.031454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080ae79b-7de9-4a0a-910d-6f3be6bf5202 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:55.043347 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080ae79b-7de9-4a0a-910d-6f3be6bf5202 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:55.048350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b41c22-d2e5-47f2-a1e0-a5133577a8e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:55.049009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b41c22-d2e5-47f2-a1e0-a5133577a8e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:55.049288 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b41c22-d2e5-47f2-a1e0-a5133577a8e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:55.049724 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b41c22-d2e5-47f2-a1e0-a5133577a8e6 admin admin] PATH_INFO: `/v3/projects/db767708d48e4cacb62f6fd397fcb27d/users/ee2099c0f4c54afaa05667840001ce33/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:55.065133 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b41c22-d2e5-47f2-a1e0-a5133577a8e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db767708d48e4cacb62f6fd397fcb27d, user_id=ee2099c0f4c54afaa05667840001ce33, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:55.067829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b41c22-d2e5-47f2-a1e0-a5133577a8e6 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:55.130762 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1082/2159] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:26:55 2022] PUT /identity/v3/projects/db767708d48e4cacb62f6fd397fcb27d/users/ee2099c0f4c54afaa05667840001ce33/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:26:55.142638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2adb60-7fd5-4b01-9cd5-7cc44652513d None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:55.142837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2adb60-7fd5-4b01-9cd5-7cc44652513d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:55.143545 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2adb60-7fd5-4b01-9cd5-7cc44652513d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:55.171538 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-cc2adb60-7fd5-4b01-9cd5-7cc44652513d None None] MFA Rules not processed for user `ee2099c0f4c54afaa05667840001ce33`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:26:55.172754 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-cc2adb60-7fd5-4b01-9cd5-7cc44652513d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:55.177811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:26:55.240662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1078/2160] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:26:55 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:26:55.252266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b41c22-d2e5-47f2-a1e0-a5133577a8e6 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:26:55.253986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-90b41c22-d2e5-47f2-a1e0-a5133577a8e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:26:55.318475 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b41c22-d2e5-47f2-a1e0-a5133577a8e6 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:26:55.322294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba70f300-2370-4e9b-89ec-321936f42e22 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:26:55.322891 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba70f300-2370-4e9b-89ec-321936f42e22 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:26:55.323834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba70f300-2370-4e9b-89ec-321936f42e22 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:26:55.324127 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba70f300-2370-4e9b-89ec-321936f42e22 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:26:55.332376 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba70f300-2370-4e9b-89ec-321936f42e22 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:26:55.333783 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba70f300-2370-4e9b-89ec-321936f42e22 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:26:55.342772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1083/2161] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:26:55 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:27:01.605271 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2adb60-7fd5-4b01-9cd5-7cc44652513d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:01.672068 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2adb60-7fd5-4b01-9cd5-7cc44652513d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:01.679129 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99efa14-4c5d-4b5d-bb7e-dfbff82b39d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:01.679929 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99efa14-4c5d-4b5d-bb7e-dfbff82b39d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:01.680342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99efa14-4c5d-4b5d-bb7e-dfbff82b39d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:01.680747 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99efa14-4c5d-4b5d-bb7e-dfbff82b39d7 admin admin] PATH_INFO: `/v3/users/ee2099c0f4c54afaa05667840001ce33` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:01.682062 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99efa14-4c5d-4b5d-bb7e-dfbff82b39d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee2099c0f4c54afaa05667840001ce33)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:01.685128 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99efa14-4c5d-4b5d-bb7e-dfbff82b39d7 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:01.882997 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a99efa14-4c5d-4b5d-bb7e-dfbff82b39d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee2099c0f4c54afaa05667840001ce33'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:01.890218 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a99efa14-4c5d-4b5d-bb7e-dfbff82b39d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee2099c0f4c54afaa05667840001ce33'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:01.890906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a99efa14-4c5d-4b5d-bb7e-dfbff82b39d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee2099c0f4c54afaa05667840001ce33'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:01.904407 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a99efa14-4c5d-4b5d-bb7e-dfbff82b39d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee2099c0f4c54afaa05667840001ce33'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:01.982963 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a99efa14-4c5d-4b5d-bb7e-dfbff82b39d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee2099c0f4c54afaa05667840001ce33'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:01.989391 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1079/2162] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:27:01 2022] DELETE /identity/v3/users/ee2099c0f4c54afaa05667840001ce33 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:27:01.994240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64cfc753-7163-49f4-836d-adbbb465f220 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:01.996461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64cfc753-7163-49f4-836d-adbbb465f220 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:01.996461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64cfc753-7163-49f4-836d-adbbb465f220 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:02.007905 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:02.014085 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-64cfc753-7163-49f4-836d-adbbb465f220 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:02.015381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-64cfc753-7163-49f4-836d-adbbb465f220 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:02.075973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1084/2163] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:02.086297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99efa14-4c5d-4b5d-bb7e-dfbff82b39d7 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:02.088448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a99efa14-4c5d-4b5d-bb7e-dfbff82b39d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:02.119866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:02.146851 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99efa14-4c5d-4b5d-bb7e-dfbff82b39d7 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:02.150637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304dc1d4-6c51-44f8-984a-8952d98644cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:02.151264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304dc1d4-6c51-44f8-984a-8952d98644cb service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:02.151778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304dc1d4-6c51-44f8-984a-8952d98644cb service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:02.152058 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304dc1d4-6c51-44f8-984a-8952d98644cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:02.161987 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304dc1d4-6c51-44f8-984a-8952d98644cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:02.163413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304dc1d4-6c51-44f8-984a-8952d98644cb service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:02.173750 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1080/2164] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:27:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:27:02.618618 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64cfc753-7163-49f4-836d-adbbb465f220 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:02.620242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-64cfc753-7163-49f4-836d-adbbb465f220 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:02.642943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64cfc753-7163-49f4-836d-adbbb465f220 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:02.665204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e18a9b-f518-44d9-be85-450d1cc79547 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:02.665871 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e18a9b-f518-44d9-be85-450d1cc79547 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:02.666147 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e18a9b-f518-44d9-be85-450d1cc79547 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:02.666424 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e18a9b-f518-44d9-be85-450d1cc79547 admin admin] PATH_INFO: `/v3/projects/db767708d48e4cacb62f6fd397fcb27d` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:02.667678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e18a9b-f518-44d9-be85-450d1cc79547 admin admin] RBAC: Authorizing `identity:delete_project(project_id=db767708d48e4cacb62f6fd397fcb27d)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:02.669784 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e18a9b-f518-44d9-be85-450d1cc79547 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:02.783220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-21e18a9b-f518-44d9-be85-450d1cc79547 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db767708d48e4cacb62f6fd397fcb27d', 'request_id': 'req-21e18a9b-f518-44d9-be85-450d1cc79547'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:02.851096 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-21e18a9b-f518-44d9-be85-450d1cc79547 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db767708d48e4cacb62f6fd397fcb27d', 'request_id': 'req-21e18a9b-f518-44d9-be85-450d1cc79547'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:02.861905 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-21e18a9b-f518-44d9-be85-450d1cc79547 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db767708d48e4cacb62f6fd397fcb27d', 'request_id': 'req-21e18a9b-f518-44d9-be85-450d1cc79547'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:02.870436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-21e18a9b-f518-44d9-be85-450d1cc79547 admin admin] The token cache is being invalidate because project db767708d48e4cacb62f6fd397fcb27d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:27:02.870726 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-21e18a9b-f518-44d9-be85-450d1cc79547 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:02.871875 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-21e18a9b-f518-44d9-be85-450d1cc79547 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:02.873441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1085/2165] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:27:02 2022] DELETE /identity/v3/projects/db767708d48e4cacb62f6fd397fcb27d => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:27:02.878480 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fa3e5e-b0db-438e-af97-701bb6ec4e04 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:02.878969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fa3e5e-b0db-438e-af97-701bb6ec4e04 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:02.879195 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fa3e5e-b0db-438e-af97-701bb6ec4e04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:02.899465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-57fa3e5e-b0db-438e-af97-701bb6ec4e04 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:02.900715 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-57fa3e5e-b0db-438e-af97-701bb6ec4e04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:02.961724 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1081/2166] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:02.969593 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a07e56-7a0a-4d4c-85a9-3b18f5f5230e None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:02.969870 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a07e56-7a0a-4d4c-85a9-3b18f5f5230e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:02.970140 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a07e56-7a0a-4d4c-85a9-3b18f5f5230e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:02.990144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-f8a07e56-7a0a-4d4c-85a9-3b18f5f5230e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:02.991327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-f8a07e56-7a0a-4d4c-85a9-3b18f5f5230e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:02.999743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1086/2167] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:27:03.010498 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fa3e5e-b0db-438e-af97-701bb6ec4e04 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:03.020688 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fa3e5e-b0db-438e-af97-701bb6ec4e04 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:03.025359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1ba238-51e5-414b-beff-1e76b670c877 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:03.025975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1ba238-51e5-414b-beff-1e76b670c877 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:03.026251 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1ba238-51e5-414b-beff-1e76b670c877 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:03.026534 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1ba238-51e5-414b-beff-1e76b670c877 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:03.027170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1ba238-51e5-414b-beff-1e76b670c877 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:27:03.027473 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1ba238-51e5-414b-beff-1e76b670c877 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:03.029085 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1ba238-51e5-414b-beff-1e76b670c877 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:03.041279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1082/2168] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:27:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:27:03.046452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a07e56-7a0a-4d4c-85a9-3b18f5f5230e None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:03.057133 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a07e56-7a0a-4d4c-85a9-3b18f5f5230e None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:03.062587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ad83b2-deb3-4554-90a9-955a39090404 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:03.063205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ad83b2-deb3-4554-90a9-955a39090404 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:03.063478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ad83b2-deb3-4554-90a9-955a39090404 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:03.063745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ad83b2-deb3-4554-90a9-955a39090404 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:03.064414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ad83b2-deb3-4554-90a9-955a39090404 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:03.066302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ad83b2-deb3-4554-90a9-955a39090404 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:03.190701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1087/2169] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:27:03 2022] POST /identity/v3/projects => generated 373 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:27:03.195046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1ba238-51e5-414b-beff-1e76b670c877 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:03.205284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1ba238-51e5-414b-beff-1e76b670c877 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:03.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d477d447-7a66-426a-9587-0c5e18e82c44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:03.264588 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d477d447-7a66-426a-9587-0c5e18e82c44 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:03.264869 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d477d447-7a66-426a-9587-0c5e18e82c44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:03.265141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d477d447-7a66-426a-9587-0c5e18e82c44 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:03.265779 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d477d447-7a66-426a-9587-0c5e18e82c44 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:03.267728 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d477d447-7a66-426a-9587-0c5e18e82c44 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:03.334383 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1083/2170] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:27:03 2022] POST /identity/v3/users => generated 287 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:27:03.338728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ad83b2-deb3-4554-90a9-955a39090404 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:03.351976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ad83b2-deb3-4554-90a9-955a39090404 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:03.357479 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5619292d-3c5b-44b5-96cd-8e7bb142538b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:03.357997 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5619292d-3c5b-44b5-96cd-8e7bb142538b admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:03.358427 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5619292d-3c5b-44b5-96cd-8e7bb142538b admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:03.358557 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5619292d-3c5b-44b5-96cd-8e7bb142538b admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:03.359230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5619292d-3c5b-44b5-96cd-8e7bb142538b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:03.360882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5619292d-3c5b-44b5-96cd-8e7bb142538b admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:03.371549 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1088/2171] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:27:03 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:27:03.375798 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d477d447-7a66-426a-9587-0c5e18e82c44 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:03.386216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d477d447-7a66-426a-9587-0c5e18e82c44 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:03.391107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbff25ec-b050-4503-bfdb-3fda91fae4ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:03.391708 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbff25ec-b050-4503-bfdb-3fda91fae4ca admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:03.391989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbff25ec-b050-4503-bfdb-3fda91fae4ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:03.392305 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbff25ec-b050-4503-bfdb-3fda91fae4ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:03.392952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbff25ec-b050-4503-bfdb-3fda91fae4ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:03.394580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbff25ec-b050-4503-bfdb-3fda91fae4ca admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:03.406310 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1084/2172] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:27:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:27:03.410747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5619292d-3c5b-44b5-96cd-8e7bb142538b admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:03.422794 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5619292d-3c5b-44b5-96cd-8e7bb142538b admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:03.428144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becfa975-39b8-4102-98fb-c46e034ee365 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:03.428813 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becfa975-39b8-4102-98fb-c46e034ee365 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:03.429092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becfa975-39b8-4102-98fb-c46e034ee365 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:03.429363 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becfa975-39b8-4102-98fb-c46e034ee365 admin admin] PATH_INFO: `/v3/projects/e727a52f046049ff8b746156a05685dc/users/a39c843eea0c47c2885e98947e58e04c/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:03.448378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-becfa975-39b8-4102-98fb-c46e034ee365 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e727a52f046049ff8b746156a05685dc, user_id=a39c843eea0c47c2885e98947e58e04c, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:03.451024 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-becfa975-39b8-4102-98fb-c46e034ee365 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:03.537914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1089/2173] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:27:03 2022] PUT /identity/v3/projects/e727a52f046049ff8b746156a05685dc/users/a39c843eea0c47c2885e98947e58e04c/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:27:03.549510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227b59c5-be39-43f0-abeb-abaefcbe813e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:03.549789 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227b59c5-be39-43f0-abeb-abaefcbe813e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:03.550057 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227b59c5-be39-43f0-abeb-abaefcbe813e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:03.579247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-227b59c5-be39-43f0-abeb-abaefcbe813e None None] MFA Rules not processed for user `a39c843eea0c47c2885e98947e58e04c`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:03.580500 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-227b59c5-be39-43f0-abeb-abaefcbe813e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:03.604124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:03.644984 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1085/2174] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:03 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:03.656228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becfa975-39b8-4102-98fb-c46e034ee365 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:03.657933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-becfa975-39b8-4102-98fb-c46e034ee365 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:03.700743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:03.726243 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becfa975-39b8-4102-98fb-c46e034ee365 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:03.735613 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76df123c-9a0a-4065-a86e-d9459753b3f4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:03.735613 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76df123c-9a0a-4065-a86e-d9459753b3f4 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:03.735613 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76df123c-9a0a-4065-a86e-d9459753b3f4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:03.736022 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76df123c-9a0a-4065-a86e-d9459753b3f4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:03.749557 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76df123c-9a0a-4065-a86e-d9459753b3f4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:03.751070 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76df123c-9a0a-4065-a86e-d9459753b3f4 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:03.760887 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1090/2175] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:27:03 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:27:13.758926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=16 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:13.961538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=14 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:18.794625 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227b59c5-be39-43f0-abeb-abaefcbe813e None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:18.796898 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-227b59c5-be39-43f0-abeb-abaefcbe813e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:18.903659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227b59c5-be39-43f0-abeb-abaefcbe813e None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:18.909824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1781e3-6867-483e-9642-ebbd1b10f8a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:18.911565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1781e3-6867-483e-9642-ebbd1b10f8a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:18.912543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1781e3-6867-483e-9642-ebbd1b10f8a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:18.919692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1781e3-6867-483e-9642-ebbd1b10f8a1 admin admin] PATH_INFO: `/v3/users/27504b8a119640b390c27decbfb72da0` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:18.919692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1781e3-6867-483e-9642-ebbd1b10f8a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=27504b8a119640b390c27decbfb72da0)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:18.919692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1781e3-6867-483e-9642-ebbd1b10f8a1 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:19.173350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-fd1781e3-6867-483e-9642-ebbd1b10f8a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '27504b8a119640b390c27decbfb72da0'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:19.181921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-fd1781e3-6867-483e-9642-ebbd1b10f8a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '27504b8a119640b390c27decbfb72da0'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:19.182616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-fd1781e3-6867-483e-9642-ebbd1b10f8a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '27504b8a119640b390c27decbfb72da0'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:19.194771 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-fd1781e3-6867-483e-9642-ebbd1b10f8a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '27504b8a119640b390c27decbfb72da0'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:19.297068 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-fd1781e3-6867-483e-9642-ebbd1b10f8a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '27504b8a119640b390c27decbfb72da0'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:19.304158 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1086/2176] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:27:18 2022] DELETE /identity/v3/users/27504b8a119640b390c27decbfb72da0 => generated 0 bytes in 510 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:27:19.308649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de161cde-1dc0-41e1-a110-414cc0e8686f None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:19.309155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de161cde-1dc0-41e1-a110-414cc0e8686f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:19.309246 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de161cde-1dc0-41e1-a110-414cc0e8686f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:19.329569 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-de161cde-1dc0-41e1-a110-414cc0e8686f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:19.330789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-de161cde-1dc0-41e1-a110-414cc0e8686f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:19.399528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1091/2177] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:19.409652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1781e3-6867-483e-9642-ebbd1b10f8a1 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:19.411079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-fd1781e3-6867-483e-9642-ebbd1b10f8a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:19.468156 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1781e3-6867-483e-9642-ebbd1b10f8a1 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:19.472693 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9587f4-99cb-4331-a797-028fd1bc26d9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:19.473221 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9587f4-99cb-4331-a797-028fd1bc26d9 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:19.476697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9587f4-99cb-4331-a797-028fd1bc26d9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:19.476697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9587f4-99cb-4331-a797-028fd1bc26d9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:19.484750 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9587f4-99cb-4331-a797-028fd1bc26d9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:19.486202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9587f4-99cb-4331-a797-028fd1bc26d9 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:19.495953 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1087/2178] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:27:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:27:19.766363 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de161cde-1dc0-41e1-a110-414cc0e8686f None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:19.769092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-de161cde-1dc0-41e1-a110-414cc0e8686f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:19.783316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de161cde-1dc0-41e1-a110-414cc0e8686f None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:19.788398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b260772c-56e1-4215-831d-21517772ff44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:19.789041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b260772c-56e1-4215-831d-21517772ff44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:19.789316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b260772c-56e1-4215-831d-21517772ff44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:19.789584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b260772c-56e1-4215-831d-21517772ff44 admin admin] PATH_INFO: `/v3/users/45c901989a7d41ceb95c68b99fa515e0` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:19.790738 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b260772c-56e1-4215-831d-21517772ff44 admin admin] RBAC: Authorizing `identity:delete_user(user_id=45c901989a7d41ceb95c68b99fa515e0)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:19.793295 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b260772c-56e1-4215-831d-21517772ff44 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:19.948452 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9587f4-99cb-4331-a797-028fd1bc26d9 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:19.950209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-6d9587f4-99cb-4331-a797-028fd1bc26d9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:19.967664 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9587f4-99cb-4331-a797-028fd1bc26d9 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:19.974691 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b5246f-4dd6-470e-86fd-c8373f901d30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:19.975627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b5246f-4dd6-470e-86fd-c8373f901d30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:19.975913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b5246f-4dd6-470e-86fd-c8373f901d30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:19.976414 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b5246f-4dd6-470e-86fd-c8373f901d30 admin admin] PATH_INFO: `/v3/projects/17245d3d23d1456fb1a2936887a75dee` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:19.977721 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b5246f-4dd6-470e-86fd-c8373f901d30 admin admin] RBAC: Authorizing `identity:delete_project(project_id=17245d3d23d1456fb1a2936887a75dee)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:19.980450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b5246f-4dd6-470e-86fd-c8373f901d30 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:20.012125 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:20.016185 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b260772c-56e1-4215-831d-21517772ff44 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '45c901989a7d41ceb95c68b99fa515e0'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:20.034325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b260772c-56e1-4215-831d-21517772ff44 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '45c901989a7d41ceb95c68b99fa515e0'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:20.055204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b260772c-56e1-4215-831d-21517772ff44 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '45c901989a7d41ceb95c68b99fa515e0'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:20.145780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-92b5246f-4dd6-470e-86fd-c8373f901d30 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '17245d3d23d1456fb1a2936887a75dee', 'request_id': 'req-92b5246f-4dd6-470e-86fd-c8373f901d30'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:20.151198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-92b5246f-4dd6-470e-86fd-c8373f901d30 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '17245d3d23d1456fb1a2936887a75dee', 'request_id': 'req-92b5246f-4dd6-470e-86fd-c8373f901d30'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:20.154956 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b260772c-56e1-4215-831d-21517772ff44 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '45c901989a7d41ceb95c68b99fa515e0'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:20.155621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b260772c-56e1-4215-831d-21517772ff44 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '45c901989a7d41ceb95c68b99fa515e0'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:20.162942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1092/2179] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:27:19 2022] DELETE /identity/v3/users/45c901989a7d41ceb95c68b99fa515e0 => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:27:20.167078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96670b3d-1904-49f4-acb1-5bc15fa73191 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:20.167637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96670b3d-1904-49f4-acb1-5bc15fa73191 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:20.168537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96670b3d-1904-49f4-acb1-5bc15fa73191 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:20.169882 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-92b5246f-4dd6-470e-86fd-c8373f901d30 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '17245d3d23d1456fb1a2936887a75dee', 'request_id': 'req-92b5246f-4dd6-470e-86fd-c8373f901d30'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:20.196222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-96670b3d-1904-49f4-acb1-5bc15fa73191 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:20.196222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-96670b3d-1904-49f4-acb1-5bc15fa73191 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:20.261089 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-92b5246f-4dd6-470e-86fd-c8373f901d30 admin admin] The token cache is being invalidate because project 17245d3d23d1456fb1a2936887a75dee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:27:20.261378 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-92b5246f-4dd6-470e-86fd-c8373f901d30 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:20.262048 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-92b5246f-4dd6-470e-86fd-c8373f901d30 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:20.263706 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1088/2180] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:27:19 2022] DELETE /identity/v3/projects/17245d3d23d1456fb1a2936887a75dee => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:27:20.272692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca963ac-66cd-488a-9e3d-9724e4e587d1 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:20.272692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca963ac-66cd-488a-9e3d-9724e4e587d1 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:20.272692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca963ac-66cd-488a-9e3d-9724e4e587d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:20.279358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1093/2181] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:27:20.290913 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96670b3d-1904-49f4-acb1-5bc15fa73191 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:20.290913 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-96670b3d-1904-49f4-acb1-5bc15fa73191 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:20.297694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-bca963ac-66cd-488a-9e3d-9724e4e587d1 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:20.299423 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-bca963ac-66cd-488a-9e3d-9724e4e587d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:20.307696 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1089/2182] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:27:20.311531 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:20.315586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded765a7-3df8-4cca-90f8-704823cba643 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:20.315911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded765a7-3df8-4cca-90f8-704823cba643 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:20.316236 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded765a7-3df8-4cca-90f8-704823cba643 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:20.335178 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-ded765a7-3df8-4cca-90f8-704823cba643 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:20.336446 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ded765a7-3df8-4cca-90f8-704823cba643 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:20.350209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1090/2183] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:27:20.354311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded765a7-3df8-4cca-90f8-704823cba643 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:20.364144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded765a7-3df8-4cca-90f8-704823cba643 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:20.365125 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96670b3d-1904-49f4-acb1-5bc15fa73191 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:20.368881 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c33525-874b-4433-8ec2-0d2b0380bc09 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:20.369024 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8864c12c-2987-4cea-aca8-77d63e7bc8ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:20.369488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c33525-874b-4433-8ec2-0d2b0380bc09 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:20.369583 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8864c12c-2987-4cea-aca8-77d63e7bc8ae admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:20.369761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c33525-874b-4433-8ec2-0d2b0380bc09 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:20.369855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8864c12c-2987-4cea-aca8-77d63e7bc8ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:20.370030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c33525-874b-4433-8ec2-0d2b0380bc09 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:20.370120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8864c12c-2987-4cea-aca8-77d63e7bc8ae admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:20.371398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8864c12c-2987-4cea-aca8-77d63e7bc8ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:27:20.371398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8864c12c-2987-4cea-aca8-77d63e7bc8ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:20.372890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8864c12c-2987-4cea-aca8-77d63e7bc8ae admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:20.381660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c33525-874b-4433-8ec2-0d2b0380bc09 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:20.383099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c33525-874b-4433-8ec2-0d2b0380bc09 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:20.392498 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1091/2184] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:27:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:27:20.393779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1094/2185] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:27:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:27:20.396555 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8864c12c-2987-4cea-aca8-77d63e7bc8ae admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:20.418933 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8864c12c-2987-4cea-aca8-77d63e7bc8ae admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:20.427395 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f2b7ce-e8f0-438c-899f-2f320d49cbb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:20.428492 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f2b7ce-e8f0-438c-899f-2f320d49cbb5 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:20.429175 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f2b7ce-e8f0-438c-899f-2f320d49cbb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:20.429704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f2b7ce-e8f0-438c-899f-2f320d49cbb5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:20.431106 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f2b7ce-e8f0-438c-899f-2f320d49cbb5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:20.434552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f2b7ce-e8f0-438c-899f-2f320d49cbb5 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:20.644213 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1092/2186] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:27:20 2022] POST /identity/v3/projects => generated 363 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:27:20.648559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c33525-874b-4433-8ec2-0d2b0380bc09 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:20.658955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c33525-874b-4433-8ec2-0d2b0380bc09 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:20.729993 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1e3079-b6b6-4883-ada9-54221b86e4d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:20.730899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1e3079-b6b6-4883-ada9-54221b86e4d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:20.731347 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1e3079-b6b6-4883-ada9-54221b86e4d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:20.731615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1e3079-b6b6-4883-ada9-54221b86e4d8 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:20.732337 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1e3079-b6b6-4883-ada9-54221b86e4d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:20.734369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1e3079-b6b6-4883-ada9-54221b86e4d8 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:20.850157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1095/2187] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:27:20 2022] POST /identity/v3/users => generated 282 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:27:20.858665 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f2b7ce-e8f0-438c-899f-2f320d49cbb5 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:20.869934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f2b7ce-e8f0-438c-899f-2f320d49cbb5 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:20.874469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986a4c8e-d4c2-4dc7-9527-b7aea0a0c4a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:20.875055 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986a4c8e-d4c2-4dc7-9527-b7aea0a0c4a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:20.875325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986a4c8e-d4c2-4dc7-9527-b7aea0a0c4a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:20.875589 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986a4c8e-d4c2-4dc7-9527-b7aea0a0c4a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:20.876284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986a4c8e-d4c2-4dc7-9527-b7aea0a0c4a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:20.877850 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986a4c8e-d4c2-4dc7-9527-b7aea0a0c4a7 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:20.887813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1093/2188] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:27:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:27:20.892215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1e3079-b6b6-4883-ada9-54221b86e4d8 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:20.902645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1e3079-b6b6-4883-ada9-54221b86e4d8 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:20.909310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b562a698-637a-44e8-bcbd-74c475c12fce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:20.909947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b562a698-637a-44e8-bcbd-74c475c12fce admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:20.910230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b562a698-637a-44e8-bcbd-74c475c12fce admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:20.910881 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b562a698-637a-44e8-bcbd-74c475c12fce admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:20.911194 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b562a698-637a-44e8-bcbd-74c475c12fce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:20.912881 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b562a698-637a-44e8-bcbd-74c475c12fce admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:20.920156 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986a4c8e-d4c2-4dc7-9527-b7aea0a0c4a7 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:20.921985 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-986a4c8e-d4c2-4dc7-9527-b7aea0a0c4a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:20.928513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1096/2189] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:27:20 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:27:20.932214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b562a698-637a-44e8-bcbd-74c475c12fce admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:20.944882 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986a4c8e-d4c2-4dc7-9527-b7aea0a0c4a7 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:20.949108 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b562a698-637a-44e8-bcbd-74c475c12fce admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:20.954595 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7e9e0e-72fa-4f54-8727-ea0b23b41d2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:20.955247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7e9e0e-72fa-4f54-8727-ea0b23b41d2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:20.955503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8d9d68-0a43-4ef6-8c69-e1d175e9a864 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:20.955614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7e9e0e-72fa-4f54-8727-ea0b23b41d2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:20.955768 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7e9e0e-72fa-4f54-8727-ea0b23b41d2a admin admin] PATH_INFO: `/v3/projects/0f6936a00a984984b27977fb51ab1eda` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:20.956203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8d9d68-0a43-4ef6-8c69-e1d175e9a864 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:20.956695 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8d9d68-0a43-4ef6-8c69-e1d175e9a864 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:20.957001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7e9e0e-72fa-4f54-8727-ea0b23b41d2a admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f6936a00a984984b27977fb51ab1eda)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:20.957151 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8d9d68-0a43-4ef6-8c69-e1d175e9a864 admin admin] PATH_INFO: `/v3/projects/dfa1623618fe43748f6a3efb03373892/users/c7359551e76d42849f51af363fcd4386/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:20.959391 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7e9e0e-72fa-4f54-8727-ea0b23b41d2a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:20.976368 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8d9d68-0a43-4ef6-8c69-e1d175e9a864 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dfa1623618fe43748f6a3efb03373892, user_id=c7359551e76d42849f51af363fcd4386, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:20.979640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8d9d68-0a43-4ef6-8c69-e1d175e9a864 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:21.064114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1097/2190] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:27:20 2022] PUT /identity/v3/projects/dfa1623618fe43748f6a3efb03373892/users/c7359551e76d42849f51af363fcd4386/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:27:21.074358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5009ca61-dff1-4c8f-b8a3-6cedf8de2576 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:21.074641 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5009ca61-dff1-4c8f-b8a3-6cedf8de2576 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:21.074912 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5009ca61-dff1-4c8f-b8a3-6cedf8de2576 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:21.104136 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-5009ca61-dff1-4c8f-b8a3-6cedf8de2576 None None] MFA Rules not processed for user `c7359551e76d42849f51af363fcd4386`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:21.105386 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-5009ca61-dff1-4c8f-b8a3-6cedf8de2576 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:21.108468 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6b7e9e0e-72fa-4f54-8727-ea0b23b41d2a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f6936a00a984984b27977fb51ab1eda', 'request_id': 'req-6b7e9e0e-72fa-4f54-8727-ea0b23b41d2a'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:21.114175 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6b7e9e0e-72fa-4f54-8727-ea0b23b41d2a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f6936a00a984984b27977fb51ab1eda', 'request_id': 'req-6b7e9e0e-72fa-4f54-8727-ea0b23b41d2a'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:21.121048 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6b7e9e0e-72fa-4f54-8727-ea0b23b41d2a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f6936a00a984984b27977fb51ab1eda', 'request_id': 'req-6b7e9e0e-72fa-4f54-8727-ea0b23b41d2a'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:21.167952 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:21.171857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1098/2191] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:21 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:27:21.183218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5009ca61-dff1-4c8f-b8a3-6cedf8de2576 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:21.195185 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-5009ca61-dff1-4c8f-b8a3-6cedf8de2576 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:21.239095 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5009ca61-dff1-4c8f-b8a3-6cedf8de2576 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:21.242756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca780042-e745-4d6b-adaa-a6521f9c1449 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:21.243348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca780042-e745-4d6b-adaa-a6521f9c1449 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:21.243618 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca780042-e745-4d6b-adaa-a6521f9c1449 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:21.243886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca780042-e745-4d6b-adaa-a6521f9c1449 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:21.252578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca780042-e745-4d6b-adaa-a6521f9c1449 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:21.253988 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca780042-e745-4d6b-adaa-a6521f9c1449 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:21.262888 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1099/2192] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:27:21 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:27:21.279605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6b7e9e0e-72fa-4f54-8727-ea0b23b41d2a admin admin] The token cache is being invalidate because project 0f6936a00a984984b27977fb51ab1eda was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:27:21.279715 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6b7e9e0e-72fa-4f54-8727-ea0b23b41d2a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:21.280376 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6b7e9e0e-72fa-4f54-8727-ea0b23b41d2a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:21.285542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1094/2193] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:27:20 2022] DELETE /identity/v3/projects/0f6936a00a984984b27977fb51ab1eda => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:27:21.287697 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd421780-820e-441a-b055-85c673b10b92 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:21.288039 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd421780-820e-441a-b055-85c673b10b92 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:21.288404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd421780-820e-441a-b055-85c673b10b92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:21.310915 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-fd421780-820e-441a-b055-85c673b10b92 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:21.312451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-fd421780-820e-441a-b055-85c673b10b92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:21.399697 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1100/2194] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:21.409046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5a9ee8-1e91-4cff-9ff3-fb1a682b794d None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:21.410045 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5a9ee8-1e91-4cff-9ff3-fb1a682b794d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:21.410419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5a9ee8-1e91-4cff-9ff3-fb1a682b794d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:21.434124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-1f5a9ee8-1e91-4cff-9ff3-fb1a682b794d None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:21.435593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-1f5a9ee8-1e91-4cff-9ff3-fb1a682b794d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:21.445794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1095/2195] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:21.450408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd421780-820e-441a-b055-85c673b10b92 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:21.462896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd421780-820e-441a-b055-85c673b10b92 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:21.468004 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa7484b-ef8f-4518-9923-b30424bf2848 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:21.468653 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa7484b-ef8f-4518-9923-b30424bf2848 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:21.468926 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa7484b-ef8f-4518-9923-b30424bf2848 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:21.469192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa7484b-ef8f-4518-9923-b30424bf2848 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:21.469860 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa7484b-ef8f-4518-9923-b30424bf2848 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:27:21.470160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa7484b-ef8f-4518-9923-b30424bf2848 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:21.471755 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa7484b-ef8f-4518-9923-b30424bf2848 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:21.488631 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1101/2196] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:27:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:27:21.495666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5a9ee8-1e91-4cff-9ff3-fb1a682b794d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:21.511638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5a9ee8-1e91-4cff-9ff3-fb1a682b794d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:21.517989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdfd202-190a-4a10-af9f-16e0bcfa2562 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:21.518709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdfd202-190a-4a10-af9f-16e0bcfa2562 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:21.519080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdfd202-190a-4a10-af9f-16e0bcfa2562 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:21.519493 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdfd202-190a-4a10-af9f-16e0bcfa2562 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:21.540502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdfd202-190a-4a10-af9f-16e0bcfa2562 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:21.540502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdfd202-190a-4a10-af9f-16e0bcfa2562 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:21.654398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=82,keystone:DELETE=14,keystone:INSERT=4 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:21.661527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1096/2197] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:27:21 2022] POST /identity/v3/projects => generated 385 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:27:21.665695 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa7484b-ef8f-4518-9923-b30424bf2848 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:21.676745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa7484b-ef8f-4518-9923-b30424bf2848 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:21.737198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6c3ff5-79e2-4fd8-badb-abf6d4424b33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:21.737805 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6c3ff5-79e2-4fd8-badb-abf6d4424b33 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:21.738083 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6c3ff5-79e2-4fd8-badb-abf6d4424b33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:21.738356 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6c3ff5-79e2-4fd8-badb-abf6d4424b33 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:21.738995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6c3ff5-79e2-4fd8-badb-abf6d4424b33 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:21.741001 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6c3ff5-79e2-4fd8-badb-abf6d4424b33 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:21.846791 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1102/2198] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:27:21 2022] POST /identity/v3/users => generated 293 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:27:21.847940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdfd202-190a-4a10-af9f-16e0bcfa2562 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:21.858046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdfd202-190a-4a10-af9f-16e0bcfa2562 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:21.863133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8477a88-7af4-4986-985e-e1eda68dd341 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:21.863727 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8477a88-7af4-4986-985e-e1eda68dd341 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:21.863999 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8477a88-7af4-4986-985e-e1eda68dd341 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:21.864293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8477a88-7af4-4986-985e-e1eda68dd341 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:21.864920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8477a88-7af4-4986-985e-e1eda68dd341 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:21.866488 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8477a88-7af4-4986-985e-e1eda68dd341 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:21.880145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1097/2199] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:27:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:27:21.885359 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6c3ff5-79e2-4fd8-badb-abf6d4424b33 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:21.892172 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6c3ff5-79e2-4fd8-badb-abf6d4424b33 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:21.897267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa9aeb7-4bf6-473f-8064-da059fc6953b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:21.897873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa9aeb7-4bf6-473f-8064-da059fc6953b admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:21.898143 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa9aeb7-4bf6-473f-8064-da059fc6953b admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:21.898412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa9aeb7-4bf6-473f-8064-da059fc6953b admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:21.899045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa9aeb7-4bf6-473f-8064-da059fc6953b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:21.900670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa9aeb7-4bf6-473f-8064-da059fc6953b admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:21.911971 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1103/2200] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:27:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:27:21.916165 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8477a88-7af4-4986-985e-e1eda68dd341 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:21.926664 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8477a88-7af4-4986-985e-e1eda68dd341 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:21.931301 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5794a469-88a7-4bc5-a74c-dfe607997a1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:21.931934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5794a469-88a7-4bc5-a74c-dfe607997a1a admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:21.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5794a469-88a7-4bc5-a74c-dfe607997a1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:21.932876 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5794a469-88a7-4bc5-a74c-dfe607997a1a admin admin] PATH_INFO: `/v3/projects/da8fc458ef6b49e18daac9fb195a2149/users/2a6687a3b09445d897879095287a8586/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:21.949063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5794a469-88a7-4bc5-a74c-dfe607997a1a admin admin] RBAC: Authorizing `identity:create_grant(project_id=da8fc458ef6b49e18daac9fb195a2149, user_id=2a6687a3b09445d897879095287a8586, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:21.951707 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5794a469-88a7-4bc5-a74c-dfe607997a1a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:22.018734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1098/2201] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:27:21 2022] PUT /identity/v3/projects/da8fc458ef6b49e18daac9fb195a2149/users/2a6687a3b09445d897879095287a8586/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:27:22.023812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634971be-d9e1-48a1-aee7-debbae6ae2c1 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:22.024121 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634971be-d9e1-48a1-aee7-debbae6ae2c1 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:22.024401 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634971be-d9e1-48a1-aee7-debbae6ae2c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:22.044257 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-634971be-d9e1-48a1-aee7-debbae6ae2c1 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:22.045453 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-634971be-d9e1-48a1-aee7-debbae6ae2c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:22.050076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:22.108749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1104/2202] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:22.119019 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5794a469-88a7-4bc5-a74c-dfe607997a1a admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:22.121313 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5794a469-88a7-4bc5-a74c-dfe607997a1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:22.182394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5794a469-88a7-4bc5-a74c-dfe607997a1a admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:22.186141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2572de-35ce-4e06-b3d7-6c189d80e1ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:22.186721 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2572de-35ce-4e06-b3d7-6c189d80e1ef service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:22.186993 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2572de-35ce-4e06-b3d7-6c189d80e1ef service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:22.190160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2572de-35ce-4e06-b3d7-6c189d80e1ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:22.197540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2572de-35ce-4e06-b3d7-6c189d80e1ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:22.200495 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2572de-35ce-4e06-b3d7-6c189d80e1ef service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:22.212065 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1099/2203] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:27:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:27:24.590527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634971be-d9e1-48a1-aee7-debbae6ae2c1 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:24.592185 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-634971be-d9e1-48a1-aee7-debbae6ae2c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:24.608030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634971be-d9e1-48a1-aee7-debbae6ae2c1 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:24.613392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f13b81-da59-4e50-91db-f614e0a110d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:24.614164 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f13b81-da59-4e50-91db-f614e0a110d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:24.614438 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f13b81-da59-4e50-91db-f614e0a110d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:24.614997 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f13b81-da59-4e50-91db-f614e0a110d3 admin admin] PATH_INFO: `/v3/users/a39c843eea0c47c2885e98947e58e04c` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:24.616429 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f13b81-da59-4e50-91db-f614e0a110d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a39c843eea0c47c2885e98947e58e04c)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:24.618769 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f13b81-da59-4e50-91db-f614e0a110d3 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:25.032801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-53f13b81-da59-4e50-91db-f614e0a110d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a39c843eea0c47c2885e98947e58e04c'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:25.042964 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-53f13b81-da59-4e50-91db-f614e0a110d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a39c843eea0c47c2885e98947e58e04c'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:25.059743 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-53f13b81-da59-4e50-91db-f614e0a110d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a39c843eea0c47c2885e98947e58e04c'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:25.181051 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-53f13b81-da59-4e50-91db-f614e0a110d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a39c843eea0c47c2885e98947e58e04c'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:25.183012 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-53f13b81-da59-4e50-91db-f614e0a110d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a39c843eea0c47c2885e98947e58e04c'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:25.193564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1105/2204] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:27:24 2022] DELETE /identity/v3/users/a39c843eea0c47c2885e98947e58e04c => generated 0 bytes in 601 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:27:25.195013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7f4dbe-6f72-4325-8bf1-15a96ccf6675 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:25.195529 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7f4dbe-6f72-4325-8bf1-15a96ccf6675 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:25.195529 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7f4dbe-6f72-4325-8bf1-15a96ccf6675 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:25.225955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-fe7f4dbe-6f72-4325-8bf1-15a96ccf6675 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:25.228774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-fe7f4dbe-6f72-4325-8bf1-15a96ccf6675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:25.371074 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1100/2205] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:25.399478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f13b81-da59-4e50-91db-f614e0a110d3 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:25.413013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-53f13b81-da59-4e50-91db-f614e0a110d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:25.534997 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f13b81-da59-4e50-91db-f614e0a110d3 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:25.543647 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253a843a-1602-41a7-9be0-84b36b2f6081 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:25.544577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253a843a-1602-41a7-9be0-84b36b2f6081 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:25.544854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253a843a-1602-41a7-9be0-84b36b2f6081 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:25.545123 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253a843a-1602-41a7-9be0-84b36b2f6081 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:25.556797 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253a843a-1602-41a7-9be0-84b36b2f6081 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:25.558467 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253a843a-1602-41a7-9be0-84b36b2f6081 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:25.571192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1106/2206] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:27:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:27:26.526266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7f4dbe-6f72-4325-8bf1-15a96ccf6675 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:26.526266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-fe7f4dbe-6f72-4325-8bf1-15a96ccf6675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:26.543085 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7f4dbe-6f72-4325-8bf1-15a96ccf6675 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:26.551226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70afec1-d446-4002-8f69-c0161fd2f9c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:26.552016 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70afec1-d446-4002-8f69-c0161fd2f9c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:26.557598 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70afec1-d446-4002-8f69-c0161fd2f9c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:26.557598 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70afec1-d446-4002-8f69-c0161fd2f9c1 admin admin] PATH_INFO: `/v3/projects/e727a52f046049ff8b746156a05685dc` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:26.557598 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70afec1-d446-4002-8f69-c0161fd2f9c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e727a52f046049ff8b746156a05685dc)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:26.557598 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70afec1-d446-4002-8f69-c0161fd2f9c1 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:26.682636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1177a9-a196-4454-980a-1844374b9e68 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:26.686547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1177a9-a196-4454-980a-1844374b9e68 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:26.686547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1177a9-a196-4454-980a-1844374b9e68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:26.703451 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f70afec1-d446-4002-8f69-c0161fd2f9c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e727a52f046049ff8b746156a05685dc', 'request_id': 'req-f70afec1-d446-4002-8f69-c0161fd2f9c1'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:26.713504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f70afec1-d446-4002-8f69-c0161fd2f9c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e727a52f046049ff8b746156a05685dc', 'request_id': 'req-f70afec1-d446-4002-8f69-c0161fd2f9c1'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:26.721772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f70afec1-d446-4002-8f69-c0161fd2f9c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e727a52f046049ff8b746156a05685dc', 'request_id': 'req-f70afec1-d446-4002-8f69-c0161fd2f9c1'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:26.724142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:26.729534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-df1177a9-a196-4454-980a-1844374b9e68 None None] MFA Rules not processed for user `d84b6d05decb47c7b55ac782d5839c1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:26.729534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-df1177a9-a196-4454-980a-1844374b9e68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:26.754120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:26.812361 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1107/2207] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:26 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:26.852198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f70afec1-d446-4002-8f69-c0161fd2f9c1 admin admin] The token cache is being invalidate because project e727a52f046049ff8b746156a05685dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:27:26.852412 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f70afec1-d446-4002-8f69-c0161fd2f9c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:26.854041 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f70afec1-d446-4002-8f69-c0161fd2f9c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:26.856072 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1101/2208] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:27:26 2022] DELETE /identity/v3/projects/e727a52f046049ff8b746156a05685dc => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:27:26.860652 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d48062-b0d5-488f-a8fe-119b13f7234e None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:26.860938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d48062-b0d5-488f-a8fe-119b13f7234e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:26.861216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d48062-b0d5-488f-a8fe-119b13f7234e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:26.883326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-69d48062-b0d5-488f-a8fe-119b13f7234e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:26.884749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-69d48062-b0d5-488f-a8fe-119b13f7234e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:26.956046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1108/2209] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:26.966302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28cba58-2f37-4f9a-985e-a0bc1fd7e940 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:26.966688 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28cba58-2f37-4f9a-985e-a0bc1fd7e940 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:26.967063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28cba58-2f37-4f9a-985e-a0bc1fd7e940 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:26.990924 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-b28cba58-2f37-4f9a-985e-a0bc1fd7e940 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:26.992386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b28cba58-2f37-4f9a-985e-a0bc1fd7e940 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:27.001790 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1102/2210] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:27.006321 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d48062-b0d5-488f-a8fe-119b13f7234e None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:27.017353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d48062-b0d5-488f-a8fe-119b13f7234e None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:27.023043 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a78d8f-ddf1-47aa-bd74-0ff3da12aec2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:27.023747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a78d8f-ddf1-47aa-bd74-0ff3da12aec2 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:27.025617 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a78d8f-ddf1-47aa-bd74-0ff3da12aec2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:27.025617 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a78d8f-ddf1-47aa-bd74-0ff3da12aec2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:27.025617 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a78d8f-ddf1-47aa-bd74-0ff3da12aec2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:27:27.025617 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a78d8f-ddf1-47aa-bd74-0ff3da12aec2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:27.027892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a78d8f-ddf1-47aa-bd74-0ff3da12aec2 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:27.044069 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1109/2211] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:27:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:27:27.045470 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28cba58-2f37-4f9a-985e-a0bc1fd7e940 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:27.055656 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28cba58-2f37-4f9a-985e-a0bc1fd7e940 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:27.060596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f627ee6a-41ae-48da-9615-fca29b5ecba3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:27.061207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f627ee6a-41ae-48da-9615-fca29b5ecba3 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:27.061486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f627ee6a-41ae-48da-9615-fca29b5ecba3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:27.061759 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f627ee6a-41ae-48da-9615-fca29b5ecba3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:27.062582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f627ee6a-41ae-48da-9615-fca29b5ecba3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:27.064466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f627ee6a-41ae-48da-9615-fca29b5ecba3 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:27.161587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1103/2212] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:27:27 2022] POST /identity/v3/projects => generated 373 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:27:27.168220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a78d8f-ddf1-47aa-bd74-0ff3da12aec2 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:27.191757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a78d8f-ddf1-47aa-bd74-0ff3da12aec2 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:27.254837 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0366f512-a52b-48a6-9743-be4d92743b41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:27.256063 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0366f512-a52b-48a6-9743-be4d92743b41 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:27.256063 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0366f512-a52b-48a6-9743-be4d92743b41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:27.256063 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0366f512-a52b-48a6-9743-be4d92743b41 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:27.256854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0366f512-a52b-48a6-9743-be4d92743b41 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:27.258877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0366f512-a52b-48a6-9743-be4d92743b41 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:27.392134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1110/2213] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:27:27 2022] POST /identity/v3/users => generated 287 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:27:27.395684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f627ee6a-41ae-48da-9615-fca29b5ecba3 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:27.405883 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f627ee6a-41ae-48da-9615-fca29b5ecba3 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:27.411449 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cdc0dd-6ccd-4aa8-9dc4-fb9b150e9bde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:27.412014 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cdc0dd-6ccd-4aa8-9dc4-fb9b150e9bde admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:27.412409 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cdc0dd-6ccd-4aa8-9dc4-fb9b150e9bde admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:27.412734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cdc0dd-6ccd-4aa8-9dc4-fb9b150e9bde admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:27.413667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5cdc0dd-6ccd-4aa8-9dc4-fb9b150e9bde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:27.415250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5cdc0dd-6ccd-4aa8-9dc4-fb9b150e9bde admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:27.577425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1104/2214] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:27:27 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:27:27.579670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0366f512-a52b-48a6-9743-be4d92743b41 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:27.594132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0366f512-a52b-48a6-9743-be4d92743b41 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:27.599955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba9678a-f279-40f1-a8e2-0a3111a771da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:27.601889 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba9678a-f279-40f1-a8e2-0a3111a771da admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:27.601889 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba9678a-f279-40f1-a8e2-0a3111a771da admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:27.601889 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba9678a-f279-40f1-a8e2-0a3111a771da admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:27.602248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba9678a-f279-40f1-a8e2-0a3111a771da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:27.604772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba9678a-f279-40f1-a8e2-0a3111a771da admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:27.618817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1111/2215] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:27:27 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:27:27.623155 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cdc0dd-6ccd-4aa8-9dc4-fb9b150e9bde admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:27.635111 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cdc0dd-6ccd-4aa8-9dc4-fb9b150e9bde admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:27.641940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16f12f3-e8f8-4bbb-9938-79d693d51e39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:27.643342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16f12f3-e8f8-4bbb-9938-79d693d51e39 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:27.643796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16f12f3-e8f8-4bbb-9938-79d693d51e39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:27.644061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16f12f3-e8f8-4bbb-9938-79d693d51e39 admin admin] PATH_INFO: `/v3/projects/e8775cdf1c9c40e8aac986896c4e0b01/users/78a9f62090244baa883ca47efed0ae20/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:27.665959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16f12f3-e8f8-4bbb-9938-79d693d51e39 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8775cdf1c9c40e8aac986896c4e0b01, user_id=78a9f62090244baa883ca47efed0ae20, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:27.671373 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16f12f3-e8f8-4bbb-9938-79d693d51e39 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:27.760590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1105/2216] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:27:27 2022] PUT /identity/v3/projects/e8775cdf1c9c40e8aac986896c4e0b01/users/78a9f62090244baa883ca47efed0ae20/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:27:27.772794 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d59967e-1dd4-4d63-86c4-3c308a93b8f7 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:27.773359 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d59967e-1dd4-4d63-86c4-3c308a93b8f7 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:27.773575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d59967e-1dd4-4d63-86c4-3c308a93b8f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:27.810939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-8d59967e-1dd4-4d63-86c4-3c308a93b8f7 None None] MFA Rules not processed for user `78a9f62090244baa883ca47efed0ae20`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:27.813303 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-8d59967e-1dd4-4d63-86c4-3c308a93b8f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:27.910062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1112/2217] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:27 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:27.924408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16f12f3-e8f8-4bbb-9938-79d693d51e39 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:27.929542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e16f12f3-e8f8-4bbb-9938-79d693d51e39 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:28.075982 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16f12f3-e8f8-4bbb-9938-79d693d51e39 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:28.087838 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8764b883-195a-4614-87af-6145fa1e9892 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:28.089405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8764b883-195a-4614-87af-6145fa1e9892 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:28.089873 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8764b883-195a-4614-87af-6145fa1e9892 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:28.090356 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8764b883-195a-4614-87af-6145fa1e9892 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:28.107208 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8764b883-195a-4614-87af-6145fa1e9892 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:28.109980 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8764b883-195a-4614-87af-6145fa1e9892 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:28.128393 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1106/2218] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:27:27 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:27:28.425636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:29.618588 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166f81c3-6e6f-45a5-ade5-fc2e030b92fc None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:29.619176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166f81c3-6e6f-45a5-ade5-fc2e030b92fc None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:29.619176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166f81c3-6e6f-45a5-ade5-fc2e030b92fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:29.650221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-166f81c3-6e6f-45a5-ade5-fc2e030b92fc None None] MFA Rules not processed for user `2a6687a3b09445d897879095287a8586`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:29.651560 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-166f81c3-6e6f-45a5-ade5-fc2e030b92fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:29.723096 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1113/2219] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:29 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:29.727443 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8764b883-195a-4614-87af-6145fa1e9892 service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:29.728854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8764b883-195a-4614-87af-6145fa1e9892 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:29.842112 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8764b883-195a-4614-87af-6145fa1e9892 service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:29.846743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83632f93-9264-42d1-86da-e754640379c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:29.847410 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83632f93-9264-42d1-86da-e754640379c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:29.847802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83632f93-9264-42d1-86da-e754640379c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:29.848158 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83632f93-9264-42d1-86da-e754640379c2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:29.848839 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83632f93-9264-42d1-86da-e754640379c2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:29.850899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83632f93-9264-42d1-86da-e754640379c2 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:29.948594 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1107/2220] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:27:29 2022] POST /identity/v3/projects => generated 385 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:27:29.952467 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166f81c3-6e6f-45a5-ade5-fc2e030b92fc None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:29.964981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166f81c3-6e6f-45a5-ade5-fc2e030b92fc None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:30.041900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dedbddb-7054-4aec-9fd9-3493213c5ac8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:30.043037 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dedbddb-7054-4aec-9fd9-3493213c5ac8 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:30.043635 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dedbddb-7054-4aec-9fd9-3493213c5ac8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:30.044146 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dedbddb-7054-4aec-9fd9-3493213c5ac8 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:30.045313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dedbddb-7054-4aec-9fd9-3493213c5ac8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:30.047986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dedbddb-7054-4aec-9fd9-3493213c5ac8 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:30.165597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1114/2221] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:27:29 2022] POST /identity/v3/users => generated 299 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:27:30.175097 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83632f93-9264-42d1-86da-e754640379c2 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:30.195820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83632f93-9264-42d1-86da-e754640379c2 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:30.206447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb13679d-71f4-4928-a6ee-5a19f4b78762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:30.207760 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb13679d-71f4-4928-a6ee-5a19f4b78762 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:30.209226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb13679d-71f4-4928-a6ee-5a19f4b78762 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:30.209748 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb13679d-71f4-4928-a6ee-5a19f4b78762 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:30.210408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb13679d-71f4-4928-a6ee-5a19f4b78762 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:30.212049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb13679d-71f4-4928-a6ee-5a19f4b78762 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:30.228198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1108/2222] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:27:30 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:27:30.234144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dedbddb-7054-4aec-9fd9-3493213c5ac8 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:30.259934 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dedbddb-7054-4aec-9fd9-3493213c5ac8 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:30.265792 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c93586a-2f0d-41c0-9dd9-52ce04ce63bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:30.266553 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c93586a-2f0d-41c0-9dd9-52ce04ce63bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:30.266916 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c93586a-2f0d-41c0-9dd9-52ce04ce63bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:30.267262 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c93586a-2f0d-41c0-9dd9-52ce04ce63bc admin admin] PATH_INFO: `/v3/projects/a7696238a2fa45afb6005154de8a14e8/users/d6fca69450794a2bb832e658fc1f6861/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:30.289046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c93586a-2f0d-41c0-9dd9-52ce04ce63bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7696238a2fa45afb6005154de8a14e8, user_id=d6fca69450794a2bb832e658fc1f6861, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:30.291913 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c93586a-2f0d-41c0-9dd9-52ce04ce63bc admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:30.417397 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1115/2223] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:27:30 2022] PUT /identity/v3/projects/a7696238a2fa45afb6005154de8a14e8/users/d6fca69450794a2bb832e658fc1f6861/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:27:30.425660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb13679d-71f4-4928-a6ee-5a19f4b78762 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:30.439536 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:INSERT=4,keystone:DELETE=1,keystone:SELECT=95 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:30.446639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb13679d-71f4-4928-a6ee-5a19f4b78762 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:30.525443 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6dd2b9-fb14-40b4-a16a-dbb0c063f411 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:30.525970 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6dd2b9-fb14-40b4-a16a-dbb0c063f411 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:30.526247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6dd2b9-fb14-40b4-a16a-dbb0c063f411 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:30.526514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6dd2b9-fb14-40b4-a16a-dbb0c063f411 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:30.527187 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6dd2b9-fb14-40b4-a16a-dbb0c063f411 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:30.528794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6dd2b9-fb14-40b4-a16a-dbb0c063f411 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:30.539449 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1109/2224] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:27:30 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:27:30.543759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c93586a-2f0d-41c0-9dd9-52ce04ce63bc admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:30.555251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c93586a-2f0d-41c0-9dd9-52ce04ce63bc admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:30.559996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8690ba8-917f-4f52-9f1a-61436818f9cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:30.560683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8690ba8-917f-4f52-9f1a-61436818f9cd admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:30.560959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8690ba8-917f-4f52-9f1a-61436818f9cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:30.561227 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8690ba8-917f-4f52-9f1a-61436818f9cd admin admin] PATH_INFO: `/v3/projects/a7696238a2fa45afb6005154de8a14e8/users/d6fca69450794a2bb832e658fc1f6861/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:30.563181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8690ba8-917f-4f52-9f1a-61436818f9cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7696238a2fa45afb6005154de8a14e8, user_id=d6fca69450794a2bb832e658fc1f6861, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:30.565937 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8690ba8-917f-4f52-9f1a-61436818f9cd admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:30.577412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1116/2225] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:27:30 2022] PUT /identity/v3/projects/a7696238a2fa45afb6005154de8a14e8/users/d6fca69450794a2bb832e658fc1f6861/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:27:31.255416 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb99f24-6986-4686-9d10-2732f2cfc9d6 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:31.256325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb99f24-6986-4686-9d10-2732f2cfc9d6 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:31.256764 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb99f24-6986-4686-9d10-2732f2cfc9d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:31.287684 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-ffb99f24-6986-4686-9d10-2732f2cfc9d6 None None] MFA Rules not processed for user `78a9f62090244baa883ca47efed0ae20`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:31.289138 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ffb99f24-6986-4686-9d10-2732f2cfc9d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:31.413146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1110/2226] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:31 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:38.315616 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c106a171-552c-4f68-9a2b-da9c1f21ad1f None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:38.316276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c106a171-552c-4f68-9a2b-da9c1f21ad1f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:38.317259 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c106a171-552c-4f68-9a2b-da9c1f21ad1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:38.336500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:38.354511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-c106a171-552c-4f68-9a2b-da9c1f21ad1f None None] MFA Rules not processed for user `d6fca69450794a2bb832e658fc1f6861`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:38.355745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c106a171-552c-4f68-9a2b-da9c1f21ad1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:38.459224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1117/2227] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:38 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:38.474490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb99f24-6986-4686-9d10-2732f2cfc9d6 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:38.477359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ffb99f24-6986-4686-9d10-2732f2cfc9d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:38.585266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb99f24-6986-4686-9d10-2732f2cfc9d6 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:38.590195 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5696b9a2-60b5-4dc3-9b68-c4a7494c43ea service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6365b2328f34ab9b1b73ececbf5cc30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:38.590884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5696b9a2-60b5-4dc3-9b68-c4a7494c43ea service glance] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:38.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5696b9a2-60b5-4dc3-9b68-c4a7494c43ea service glance] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:38.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5696b9a2-60b5-4dc3-9b68-c4a7494c43ea service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:38.683480 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5696b9a2-60b5-4dc3-9b68-c4a7494c43ea service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:38.685025 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5696b9a2-60b5-4dc3-9b68-c4a7494c43ea service glance] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:38.690484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:38.706318 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1111/2228] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:27:38 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:27:39.063600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64538b76-29ef-4797-8627-ef7aacd2d0d2 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:39.063889 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64538b76-29ef-4797-8627-ef7aacd2d0d2 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:39.064194 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64538b76-29ef-4797-8627-ef7aacd2d0d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:39.084685 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-64538b76-29ef-4797-8627-ef7aacd2d0d2 None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:39.085903 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-64538b76-29ef-4797-8627-ef7aacd2d0d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:39.146746 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1118/2229] 10.222.0.47 () {58 vars in 1009 bytes} [Sun Aug 21 01:27:39 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:27:39.191136 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5696b9a2-60b5-4dc3-9b68-c4a7494c43ea service glance] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:39.192836 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5696b9a2-60b5-4dc3-9b68-c4a7494c43ea service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:39.261590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5696b9a2-60b5-4dc3-9b68-c4a7494c43ea service glance] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:39.266725 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756e690c-e21e-4fb1-9c2b-0f1db431827b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:39.267325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756e690c-e21e-4fb1-9c2b-0f1db431827b service swift] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:39.267599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756e690c-e21e-4fb1-9c2b-0f1db431827b service swift] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:39.267891 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756e690c-e21e-4fb1-9c2b-0f1db431827b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:39.277551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756e690c-e21e-4fb1-9c2b-0f1db431827b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:39.279125 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756e690c-e21e-4fb1-9c2b-0f1db431827b service swift] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:39.287377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1112/2230] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:27:39 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 21 01:27:44.304226 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d57bad-acdc-498a-a44a-7f94dca85a78 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:44.306968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d57bad-acdc-498a-a44a-7f94dca85a78 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:44.306968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d57bad-acdc-498a-a44a-7f94dca85a78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:44.340656 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-04d57bad-acdc-498a-a44a-7f94dca85a78 None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:44.355636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-04d57bad-acdc-498a-a44a-7f94dca85a78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:44.370198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1119/2231] 10.222.0.47 () {58 vars in 1009 bytes} [Sun Aug 21 01:27:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:44.384781 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756e690c-e21e-4fb1-9c2b-0f1db431827b service swift] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:44.396830 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756e690c-e21e-4fb1-9c2b-0f1db431827b service swift] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:44.405309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457ab7ac-b7d9-40fa-81b1-08e0159eb708 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:44.406046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457ab7ac-b7d9-40fa-81b1-08e0159eb708 service swift] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:44.406511 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457ab7ac-b7d9-40fa-81b1-08e0159eb708 service swift] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:44.406963 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457ab7ac-b7d9-40fa-81b1-08e0159eb708 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:44.421904 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457ab7ac-b7d9-40fa-81b1-08e0159eb708 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:44.425840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457ab7ac-b7d9-40fa-81b1-08e0159eb708 service swift] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:44.441207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1113/2232] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:27:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 21 01:27:50.636207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d57bad-acdc-498a-a44a-7f94dca85a78 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:50.679324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-04d57bad-acdc-498a-a44a-7f94dca85a78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:50.826424 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d57bad-acdc-498a-a44a-7f94dca85a78 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:50.833242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3199ab6e-4952-4311-b7ed-2fd6415e68db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:50.834002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3199ab6e-4952-4311-b7ed-2fd6415e68db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:50.834277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3199ab6e-4952-4311-b7ed-2fd6415e68db admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:50.834551 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3199ab6e-4952-4311-b7ed-2fd6415e68db admin admin] PATH_INFO: `/v3/users/01ffe9b26c36459fafa185e0e51b2f65` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:50.836375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3199ab6e-4952-4311-b7ed-2fd6415e68db admin admin] RBAC: Authorizing `identity:delete_user(user_id=01ffe9b26c36459fafa185e0e51b2f65)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:50.840015 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3199ab6e-4952-4311-b7ed-2fd6415e68db admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:51.361195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:51.395128 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3199ab6e-4952-4311-b7ed-2fd6415e68db admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '01ffe9b26c36459fafa185e0e51b2f65'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:51.418074 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3199ab6e-4952-4311-b7ed-2fd6415e68db admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '01ffe9b26c36459fafa185e0e51b2f65'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:51.449881 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3199ab6e-4952-4311-b7ed-2fd6415e68db admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '01ffe9b26c36459fafa185e0e51b2f65'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:51.538877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3199ab6e-4952-4311-b7ed-2fd6415e68db admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '01ffe9b26c36459fafa185e0e51b2f65'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:51.541008 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3199ab6e-4952-4311-b7ed-2fd6415e68db admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '01ffe9b26c36459fafa185e0e51b2f65'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:51.574393 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1120/2233] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:27:50 2022] DELETE /identity/v3/users/01ffe9b26c36459fafa185e0e51b2f65 => generated 0 bytes in 938 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:27:51.575940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98725d92-077a-43f5-bce8-86807b49da6d None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:51.577273 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98725d92-077a-43f5-bce8-86807b49da6d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:51.577273 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98725d92-077a-43f5-bce8-86807b49da6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:51.622073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-98725d92-077a-43f5-bce8-86807b49da6d None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:51.623674 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-98725d92-077a-43f5-bce8-86807b49da6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:51.800455 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1114/2234] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:51.810807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3199ab6e-4952-4311-b7ed-2fd6415e68db admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:51.812433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3199ab6e-4952-4311-b7ed-2fd6415e68db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:51.960962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3199ab6e-4952-4311-b7ed-2fd6415e68db admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:51.966452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706f0c83-b147-42be-a5d4-65ef20c6fe91 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:51.967157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706f0c83-b147-42be-a5d4-65ef20c6fe91 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:51.967787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706f0c83-b147-42be-a5d4-65ef20c6fe91 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:51.967957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706f0c83-b147-42be-a5d4-65ef20c6fe91 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:52.017802 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706f0c83-b147-42be-a5d4-65ef20c6fe91 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:52.020303 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706f0c83-b147-42be-a5d4-65ef20c6fe91 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:52.040709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1121/2235] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:27:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:27:53.597216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98725d92-077a-43f5-bce8-86807b49da6d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:53.598707 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-98725d92-077a-43f5-bce8-86807b49da6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:53.618606 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98725d92-077a-43f5-bce8-86807b49da6d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:53.630956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45140f1a-dd01-4b46-8e03-229c835e1bc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:53.632262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45140f1a-dd01-4b46-8e03-229c835e1bc3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:53.632609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45140f1a-dd01-4b46-8e03-229c835e1bc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:53.632925 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45140f1a-dd01-4b46-8e03-229c835e1bc3 admin admin] PATH_INFO: `/v3/projects/747410d56cdf42da95c8d1282d3d5d06` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:53.636855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45140f1a-dd01-4b46-8e03-229c835e1bc3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=747410d56cdf42da95c8d1282d3d5d06)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:53.638881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45140f1a-dd01-4b46-8e03-229c835e1bc3 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:53.873873 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-45140f1a-dd01-4b46-8e03-229c835e1bc3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '747410d56cdf42da95c8d1282d3d5d06', 'request_id': 'req-45140f1a-dd01-4b46-8e03-229c835e1bc3'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:53.903015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-45140f1a-dd01-4b46-8e03-229c835e1bc3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '747410d56cdf42da95c8d1282d3d5d06', 'request_id': 'req-45140f1a-dd01-4b46-8e03-229c835e1bc3'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:53.919197 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-45140f1a-dd01-4b46-8e03-229c835e1bc3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '747410d56cdf42da95c8d1282d3d5d06', 'request_id': 'req-45140f1a-dd01-4b46-8e03-229c835e1bc3'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:54.091745 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:54.098626 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-45140f1a-dd01-4b46-8e03-229c835e1bc3 admin admin] The token cache is being invalidate because project 747410d56cdf42da95c8d1282d3d5d06 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:27:54.099150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-45140f1a-dd01-4b46-8e03-229c835e1bc3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:54.100738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-45140f1a-dd01-4b46-8e03-229c835e1bc3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:27:54.107823 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4343c61-676d-49f0-aa44-773529809f19 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:54.109017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4343c61-676d-49f0-aa44-773529809f19 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:54.112260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4343c61-676d-49f0-aa44-773529809f19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:54.114996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1115/2236] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:27:53 2022] DELETE /identity/v3/projects/747410d56cdf42da95c8d1282d3d5d06 => generated 0 bytes in 506 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:27:54.149703 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-a4343c61-676d-49f0-aa44-773529809f19 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:54.150938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-a4343c61-676d-49f0-aa44-773529809f19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:54.390922 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1122/2237] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 284 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:54.400163 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e311bc-a8db-4bc8-bb73-5834fdc19be8 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:54.400449 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e311bc-a8db-4bc8-bb73-5834fdc19be8 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:54.400715 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e311bc-a8db-4bc8-bb73-5834fdc19be8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:54.458756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-19e311bc-a8db-4bc8-bb73-5834fdc19be8 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:54.460019 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-19e311bc-a8db-4bc8-bb73-5834fdc19be8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:54.481772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1116/2238] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:54.486964 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4343c61-676d-49f0-aa44-773529809f19 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:54.503846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4343c61-676d-49f0-aa44-773529809f19 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:54.520001 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55054033-4fd2-45e9-8aeb-39dc87e81a50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:54.522910 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55054033-4fd2-45e9-8aeb-39dc87e81a50 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:54.522910 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55054033-4fd2-45e9-8aeb-39dc87e81a50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:54.522910 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55054033-4fd2-45e9-8aeb-39dc87e81a50 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:54.524012 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55054033-4fd2-45e9-8aeb-39dc87e81a50 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:27:54.524012 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55054033-4fd2-45e9-8aeb-39dc87e81a50 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:54.531608 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55054033-4fd2-45e9-8aeb-39dc87e81a50 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:54.568092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1123/2239] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:27:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:27:54.569298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e311bc-a8db-4bc8-bb73-5834fdc19be8 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:54.582117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e311bc-a8db-4bc8-bb73-5834fdc19be8 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:54.592336 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859c7e7f-8bc7-4968-bf84-f4ace0f6bc22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:54.592960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859c7e7f-8bc7-4968-bf84-f4ace0f6bc22 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:54.593232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859c7e7f-8bc7-4968-bf84-f4ace0f6bc22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:54.593497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859c7e7f-8bc7-4968-bf84-f4ace0f6bc22 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:54.594312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859c7e7f-8bc7-4968-bf84-f4ace0f6bc22 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:54.596245 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859c7e7f-8bc7-4968-bf84-f4ace0f6bc22 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:54.758560 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1117/2240] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:27:54 2022] POST /identity/v3/projects => generated 399 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:27:54.767137 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55054033-4fd2-45e9-8aeb-39dc87e81a50 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:54.804397 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55054033-4fd2-45e9-8aeb-39dc87e81a50 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:54.917407 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:27:54.952093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8443782-02b9-4fd7-a7d0-25133b652c06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:54.953759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8443782-02b9-4fd7-a7d0-25133b652c06 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:54.955103 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8443782-02b9-4fd7-a7d0-25133b652c06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:54.955870 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8443782-02b9-4fd7-a7d0-25133b652c06 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:54.958408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8443782-02b9-4fd7-a7d0-25133b652c06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:54.961715 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8443782-02b9-4fd7-a7d0-25133b652c06 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:55.284359 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1124/2241] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:27:54 2022] POST /identity/v3/users => generated 300 bytes in 517 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:27:55.285109 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859c7e7f-8bc7-4968-bf84-f4ace0f6bc22 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:55.305154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859c7e7f-8bc7-4968-bf84-f4ace0f6bc22 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:55.313676 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56da1ed-f649-4690-ba6f-cac231ee197a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:55.314805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56da1ed-f649-4690-ba6f-cac231ee197a admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:55.315289 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56da1ed-f649-4690-ba6f-cac231ee197a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:55.315728 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56da1ed-f649-4690-ba6f-cac231ee197a admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:55.316640 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56da1ed-f649-4690-ba6f-cac231ee197a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:55.319687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56da1ed-f649-4690-ba6f-cac231ee197a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:55.336548 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1118/2242] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:27:55 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:27:55.342265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8443782-02b9-4fd7-a7d0-25133b652c06 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:55.371999 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8443782-02b9-4fd7-a7d0-25133b652c06 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:55.378054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da86879d-436f-447c-90fc-46231c5456dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:55.379645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da86879d-436f-447c-90fc-46231c5456dd admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:55.380081 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da86879d-436f-447c-90fc-46231c5456dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:55.380434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da86879d-436f-447c-90fc-46231c5456dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:55.381126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da86879d-436f-447c-90fc-46231c5456dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:55.383844 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da86879d-436f-447c-90fc-46231c5456dd admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:55.402181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1125/2243] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:27:55 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:27:55.406048 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56da1ed-f649-4690-ba6f-cac231ee197a admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:55.422000 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56da1ed-f649-4690-ba6f-cac231ee197a admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:55.431309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99aa0eed-a3a1-4231-971e-4b0e72c8009e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:55.432650 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99aa0eed-a3a1-4231-971e-4b0e72c8009e admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:55.433126 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99aa0eed-a3a1-4231-971e-4b0e72c8009e admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:55.434018 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99aa0eed-a3a1-4231-971e-4b0e72c8009e admin admin] PATH_INFO: `/v3/projects/a201641ad3694794a5cbfc84e0b5eefc/users/4f4c145738f941c4ad7cb225187a010c/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:55.471358 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99aa0eed-a3a1-4231-971e-4b0e72c8009e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a201641ad3694794a5cbfc84e0b5eefc, user_id=4f4c145738f941c4ad7cb225187a010c, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:55.475917 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99aa0eed-a3a1-4231-971e-4b0e72c8009e admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:55.585078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1119/2244] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:27:55 2022] PUT /identity/v3/projects/a201641ad3694794a5cbfc84e0b5eefc/users/4f4c145738f941c4ad7cb225187a010c/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:27:55.591143 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8021693d-53c4-420a-82d0-c0b089a84bef None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:55.591478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8021693d-53c4-420a-82d0-c0b089a84bef None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:55.591765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8021693d-53c4-420a-82d0-c0b089a84bef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:55.654521 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-8021693d-53c4-420a-82d0-c0b089a84bef None None] MFA Rules not processed for user `4f4c145738f941c4ad7cb225187a010c`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:55.655767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-8021693d-53c4-420a-82d0-c0b089a84bef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:55.832310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1126/2245] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:55 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:27:55.845141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99aa0eed-a3a1-4231-971e-4b0e72c8009e admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:27:55.847803 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-99aa0eed-a3a1-4231-971e-4b0e72c8009e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:56.082372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99aa0eed-a3a1-4231-971e-4b0e72c8009e admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:27:56.099183 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0ccbf0-f7f6-4415-b00c-e489f72a9d3b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:27:56.100323 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0ccbf0-f7f6-4415-b00c-e489f72a9d3b service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:56.100745 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0ccbf0-f7f6-4415-b00c-e489f72a9d3b service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:56.101159 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0ccbf0-f7f6-4415-b00c-e489f72a9d3b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:56.137793 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0ccbf0-f7f6-4415-b00c-e489f72a9d3b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:27:56.139749 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0ccbf0-f7f6-4415-b00c-e489f72a9d3b service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:27:56.167522 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1120/2246] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:27:55 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:27:57.894273 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ef4c6d-1081-49a2-8382-f9e42cdfb709 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:27:57.895300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ef4c6d-1081-49a2-8382-f9e42cdfb709 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:27:57.896077 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ef4c6d-1081-49a2-8382-f9e42cdfb709 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:27:58.009715 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-a2ef4c6d-1081-49a2-8382-f9e42cdfb709 None None] MFA Rules not processed for user `2a6687a3b09445d897879095287a8586`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:27:58.013011 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-a2ef4c6d-1081-49a2-8382-f9e42cdfb709 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:27:58.203484 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1127/2247] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:27:57 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:06.149324 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=53,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:06.621005 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0ccbf0-f7f6-4415-b00c-e489f72a9d3b service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:06.687929 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0ccbf0-f7f6-4415-b00c-e489f72a9d3b service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:06.693088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccf60f4-8b65-4e71-8574-d2e58b60a8bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:06.696719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccf60f4-8b65-4e71-8574-d2e58b60a8bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:06.696719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccf60f4-8b65-4e71-8574-d2e58b60a8bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:06.696719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccf60f4-8b65-4e71-8574-d2e58b60a8bc admin admin] PATH_INFO: `/v3/users/4f4c145738f941c4ad7cb225187a010c` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:06.696719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccf60f4-8b65-4e71-8574-d2e58b60a8bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=4f4c145738f941c4ad7cb225187a010c)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:06.697948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccf60f4-8b65-4e71-8574-d2e58b60a8bc admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:06.778029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f8b5e6-e18e-48d5-969f-2144f5938f76 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:06.778568 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f8b5e6-e18e-48d5-969f-2144f5938f76 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:06.778568 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f8b5e6-e18e-48d5-969f-2144f5938f76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:06.799096 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-04f8b5e6-e18e-48d5-969f-2144f5938f76 None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:06.800337 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-04f8b5e6-e18e-48d5-969f-2144f5938f76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:06.855072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:06.881254 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1128/2248] 10.222.0.47 () {58 vars in 1009 bytes} [Sun Aug 21 01:28:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:28:06.881254 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f8b5e6-e18e-48d5-969f-2144f5938f76 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:06.882360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-04f8b5e6-e18e-48d5-969f-2144f5938f76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:06.952400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-8ccf60f4-8b65-4e71-8574-d2e58b60a8bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4f4c145738f941c4ad7cb225187a010c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:06.964996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-8ccf60f4-8b65-4e71-8574-d2e58b60a8bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4f4c145738f941c4ad7cb225187a010c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:06.965770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-8ccf60f4-8b65-4e71-8574-d2e58b60a8bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4f4c145738f941c4ad7cb225187a010c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:06.971663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f8b5e6-e18e-48d5-969f-2144f5938f76 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:06.977773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184f19f6-220b-4b33-b453-10009a0e8cfc service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:06.979476 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184f19f6-220b-4b33-b453-10009a0e8cfc service swift] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:06.979939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184f19f6-220b-4b33-b453-10009a0e8cfc service swift] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:06.980326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184f19f6-220b-4b33-b453-10009a0e8cfc service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:06.980543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-8ccf60f4-8b65-4e71-8574-d2e58b60a8bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4f4c145738f941c4ad7cb225187a010c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:06.990722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184f19f6-220b-4b33-b453-10009a0e8cfc service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:06.994551 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184f19f6-220b-4b33-b453-10009a0e8cfc service swift] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:07.003989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1129/2249] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:28:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 21 01:28:07.071189 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-8ccf60f4-8b65-4e71-8574-d2e58b60a8bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4f4c145738f941c4ad7cb225187a010c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:07.077653 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1121/2250] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:28:06 2022] DELETE /identity/v3/users/4f4c145738f941c4ad7cb225187a010c => generated 0 bytes in 457 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:28:07.082196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aeba3c-0dbc-4cb8-9cf0-3541d29a22fa None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:07.082478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aeba3c-0dbc-4cb8-9cf0-3541d29a22fa None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:07.082752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aeba3c-0dbc-4cb8-9cf0-3541d29a22fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:07.103532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-10aeba3c-0dbc-4cb8-9cf0-3541d29a22fa None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:07.104837 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-10aeba3c-0dbc-4cb8-9cf0-3541d29a22fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:07.168153 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1130/2251] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:28:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:07.177981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccf60f4-8b65-4e71-8574-d2e58b60a8bc admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:07.180912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8ccf60f4-8b65-4e71-8574-d2e58b60a8bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:07.240457 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccf60f4-8b65-4e71-8574-d2e58b60a8bc admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:07.244678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b4a1f5-4d7b-4cef-be7f-8a67c5e8bcc4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:07.245284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b4a1f5-4d7b-4cef-be7f-8a67c5e8bcc4 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:07.245560 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b4a1f5-4d7b-4cef-be7f-8a67c5e8bcc4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:07.245833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b4a1f5-4d7b-4cef-be7f-8a67c5e8bcc4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:07.255925 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b4a1f5-4d7b-4cef-be7f-8a67c5e8bcc4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:07.257427 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b4a1f5-4d7b-4cef-be7f-8a67c5e8bcc4 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:07.272225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1122/2252] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:28:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:28:07.844595 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d33f95b-f7c6-4fc4-9640-a2bc8df4da02 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:07.845142 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d33f95b-f7c6-4fc4-9640-a2bc8df4da02 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:07.846063 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d33f95b-f7c6-4fc4-9640-a2bc8df4da02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:07.884774 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-1d33f95b-f7c6-4fc4-9640-a2bc8df4da02 None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:07.886433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1d33f95b-f7c6-4fc4-9640-a2bc8df4da02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:07.980004 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1131/2253] 10.222.0.47 () {58 vars in 1009 bytes} [Sun Aug 21 01:28:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:07.994514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b4a1f5-4d7b-4cef-be7f-8a67c5e8bcc4 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:07.996210 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e9b4a1f5-4d7b-4cef-be7f-8a67c5e8bcc4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:08.026626 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:08.076196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b4a1f5-4d7b-4cef-be7f-8a67c5e8bcc4 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:08.081398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9bf12f-76cb-44e2-993c-aa77f54ba68e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:08.082012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9bf12f-76cb-44e2-993c-aa77f54ba68e service swift] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:08.082283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9bf12f-76cb-44e2-993c-aa77f54ba68e service swift] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:08.082551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9bf12f-76cb-44e2-993c-aa77f54ba68e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:08.091931 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9bf12f-76cb-44e2-993c-aa77f54ba68e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:08.093549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9bf12f-76cb-44e2-993c-aa77f54ba68e service swift] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:08.103761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1123/2254] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:28:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 21 01:28:08.205548 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d33f95b-f7c6-4fc4-9640-a2bc8df4da02 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:08.207135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1d33f95b-f7c6-4fc4-9640-a2bc8df4da02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:08.223808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d33f95b-f7c6-4fc4-9640-a2bc8df4da02 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:08.228591 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ad2c81-ecbf-41f3-b591-2790ab7dbc34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:08.229240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ad2c81-ecbf-41f3-b591-2790ab7dbc34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:08.229508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ad2c81-ecbf-41f3-b591-2790ab7dbc34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:08.230459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ad2c81-ecbf-41f3-b591-2790ab7dbc34 admin admin] PATH_INFO: `/v3/projects/a201641ad3694794a5cbfc84e0b5eefc` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:08.231676 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ad2c81-ecbf-41f3-b591-2790ab7dbc34 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a201641ad3694794a5cbfc84e0b5eefc)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:08.237657 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ad2c81-ecbf-41f3-b591-2790ab7dbc34 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:08.357210 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:08.394960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-74ad2c81-ecbf-41f3-b591-2790ab7dbc34 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a201641ad3694794a5cbfc84e0b5eefc', 'request_id': 'req-74ad2c81-ecbf-41f3-b591-2790ab7dbc34'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:08.514261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-74ad2c81-ecbf-41f3-b591-2790ab7dbc34 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a201641ad3694794a5cbfc84e0b5eefc', 'request_id': 'req-74ad2c81-ecbf-41f3-b591-2790ab7dbc34'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:08.532630 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-74ad2c81-ecbf-41f3-b591-2790ab7dbc34 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a201641ad3694794a5cbfc84e0b5eefc', 'request_id': 'req-74ad2c81-ecbf-41f3-b591-2790ab7dbc34'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:08.575945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-74ad2c81-ecbf-41f3-b591-2790ab7dbc34 admin admin] The token cache is being invalidate because project a201641ad3694794a5cbfc84e0b5eefc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:28:08.579857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-74ad2c81-ecbf-41f3-b591-2790ab7dbc34 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:08.579857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-74ad2c81-ecbf-41f3-b591-2790ab7dbc34 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:08.584893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1132/2255] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:28:08 2022] DELETE /identity/v3/projects/a201641ad3694794a5cbfc84e0b5eefc => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:28:08.591743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a49346-7159-4f8e-971a-cd0084871ca0 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:08.592631 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a49346-7159-4f8e-971a-cd0084871ca0 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:08.593173 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a49346-7159-4f8e-971a-cd0084871ca0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:08.640746 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-52a49346-7159-4f8e-971a-cd0084871ca0 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:08.642090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-52a49346-7159-4f8e-971a-cd0084871ca0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:08.752580 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1124/2256] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:28:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:08.763327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f2ff9d-5f6f-4d22-93d3-616a43768c5e None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:08.763327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f2ff9d-5f6f-4d22-93d3-616a43768c5e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:08.763327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f2ff9d-5f6f-4d22-93d3-616a43768c5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:08.806924 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-e9f2ff9d-5f6f-4d22-93d3-616a43768c5e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:08.808473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e9f2ff9d-5f6f-4d22-93d3-616a43768c5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:08.820303 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1133/2257] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:28:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:08.824030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a49346-7159-4f8e-971a-cd0084871ca0 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:08.843160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a49346-7159-4f8e-971a-cd0084871ca0 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:08.854064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7939e4-ccfd-4ba3-aa1f-2ae5c502f3d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:08.854959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7939e4-ccfd-4ba3-aa1f-2ae5c502f3d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:08.855572 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7939e4-ccfd-4ba3-aa1f-2ae5c502f3d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:08.856284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7939e4-ccfd-4ba3-aa1f-2ae5c502f3d8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:08.858297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7939e4-ccfd-4ba3-aa1f-2ae5c502f3d8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:28:08.859153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7939e4-ccfd-4ba3-aa1f-2ae5c502f3d8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:08.861820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7939e4-ccfd-4ba3-aa1f-2ae5c502f3d8 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:08.879660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1125/2258] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:28:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:28:08.884993 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f2ff9d-5f6f-4d22-93d3-616a43768c5e None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:08.904513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f2ff9d-5f6f-4d22-93d3-616a43768c5e None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:08.912175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cc38e4-eb66-43fc-9f8d-ba7982ec6213 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:08.913127 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cc38e4-eb66-43fc-9f8d-ba7982ec6213 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:08.913576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cc38e4-eb66-43fc-9f8d-ba7982ec6213 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:08.914008 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cc38e4-eb66-43fc-9f8d-ba7982ec6213 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:08.917879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cc38e4-eb66-43fc-9f8d-ba7982ec6213 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:08.920223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cc38e4-eb66-43fc-9f8d-ba7982ec6213 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:09.036496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1134/2259] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:28:08 2022] POST /identity/v3/projects => generated 371 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:28:09.040302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7939e4-ccfd-4ba3-aa1f-2ae5c502f3d8 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:09.052864 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7939e4-ccfd-4ba3-aa1f-2ae5c502f3d8 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:09.189559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc21ba29-15fd-4d3b-959a-a7abd8fcf980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:09.190183 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc21ba29-15fd-4d3b-959a-a7abd8fcf980 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:09.190456 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc21ba29-15fd-4d3b-959a-a7abd8fcf980 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:09.190853 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc21ba29-15fd-4d3b-959a-a7abd8fcf980 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:09.191776 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc21ba29-15fd-4d3b-959a-a7abd8fcf980 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:09.194692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc21ba29-15fd-4d3b-959a-a7abd8fcf980 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:09.495820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1126/2260] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:28:09 2022] POST /identity/v3/users => generated 286 bytes in 456 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:28:09.503943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cc38e4-eb66-43fc-9f8d-ba7982ec6213 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:09.517424 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cc38e4-eb66-43fc-9f8d-ba7982ec6213 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:09.524787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0348bf0-d93b-4b1d-b2a6-5fee7e686e79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:09.524787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0348bf0-d93b-4b1d-b2a6-5fee7e686e79 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:09.524787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0348bf0-d93b-4b1d-b2a6-5fee7e686e79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:09.524787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0348bf0-d93b-4b1d-b2a6-5fee7e686e79 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:09.525265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0348bf0-d93b-4b1d-b2a6-5fee7e686e79 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:09.526112 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0348bf0-d93b-4b1d-b2a6-5fee7e686e79 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:09.542414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1135/2261] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:28:09 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:28:09.546440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc21ba29-15fd-4d3b-959a-a7abd8fcf980 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:09.558233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc21ba29-15fd-4d3b-959a-a7abd8fcf980 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:09.569825 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a17cab-6682-4090-9038-b4de31a405ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:09.570448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a17cab-6682-4090-9038-b4de31a405ff admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:09.570769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a17cab-6682-4090-9038-b4de31a405ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:09.571065 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a17cab-6682-4090-9038-b4de31a405ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:09.571731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a17cab-6682-4090-9038-b4de31a405ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:09.573508 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a17cab-6682-4090-9038-b4de31a405ff admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:09.585217 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1127/2262] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:28:09 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:28:09.590448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0348bf0-d93b-4b1d-b2a6-5fee7e686e79 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:09.601650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0348bf0-d93b-4b1d-b2a6-5fee7e686e79 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:09.607818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be680be-b0d0-4f8c-8cda-1e4205ccbc2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:09.608763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be680be-b0d0-4f8c-8cda-1e4205ccbc2d admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:09.609037 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be680be-b0d0-4f8c-8cda-1e4205ccbc2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:09.609306 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be680be-b0d0-4f8c-8cda-1e4205ccbc2d admin admin] PATH_INFO: `/v3/projects/ac06ecc6fc93477ba4909284525df514/users/03d34d1e66404d259599806d040628ef/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:09.628291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be680be-b0d0-4f8c-8cda-1e4205ccbc2d admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac06ecc6fc93477ba4909284525df514, user_id=03d34d1e66404d259599806d040628ef, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:09.631408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be680be-b0d0-4f8c-8cda-1e4205ccbc2d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:09.822529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1136/2263] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:28:09 2022] PUT /identity/v3/projects/ac06ecc6fc93477ba4909284525df514/users/03d34d1e66404d259599806d040628ef/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:28:09.828281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2361e0c2-bd6e-416e-a132-f527a13c19a4 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:09.829186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2361e0c2-bd6e-416e-a132-f527a13c19a4 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:09.829474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2361e0c2-bd6e-416e-a132-f527a13c19a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:09.854620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-2361e0c2-bd6e-416e-a132-f527a13c19a4 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:09.855871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-2361e0c2-bd6e-416e-a132-f527a13c19a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:09.869398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:09.930369 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1128/2264] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:28:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:09.940459 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be680be-b0d0-4f8c-8cda-1e4205ccbc2d admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:09.942663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9be680be-b0d0-4f8c-8cda-1e4205ccbc2d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:10.032032 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be680be-b0d0-4f8c-8cda-1e4205ccbc2d admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:10.036330 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698703a7-508b-477c-849c-e55a1502a2fc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:10.039353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698703a7-508b-477c-849c-e55a1502a2fc service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:10.039353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698703a7-508b-477c-849c-e55a1502a2fc service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:10.039353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698703a7-508b-477c-849c-e55a1502a2fc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:10.047943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698703a7-508b-477c-849c-e55a1502a2fc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:10.049489 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698703a7-508b-477c-849c-e55a1502a2fc service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:10.062397 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1137/2265] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:28:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:28:10.968372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4c776c-b973-467e-9ae9-ea9599918f7f None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:10.968944 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4c776c-b973-467e-9ae9-ea9599918f7f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:10.968944 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4c776c-b973-467e-9ae9-ea9599918f7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:10.989638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-ce4c776c-b973-467e-9ae9-ea9599918f7f None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:10.990858 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ce4c776c-b973-467e-9ae9-ea9599918f7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:11.085843 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1129/2266] 10.222.0.47 () {58 vars in 967 bytes} [Sun Aug 21 01:28:10 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:11.102804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698703a7-508b-477c-849c-e55a1502a2fc service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:11.105344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-698703a7-508b-477c-849c-e55a1502a2fc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:11.152789 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5113a2-d143-4c9e-9a9d-50028a950eb2 None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:11.153276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5113a2-d143-4c9e-9a9d-50028a950eb2 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:11.154222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5113a2-d143-4c9e-9a9d-50028a950eb2 None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:11.156638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1130/2267] 10.222.0.47 () {54 vars in 921 bytes} [Sun Aug 21 01:28:11 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 21 01:28:11.167098 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec7587c-8c4b-46c3-86d3-a002ec7a5cac None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:11.167524 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec7587c-8c4b-46c3-86d3-a002ec7a5cac None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:11.167796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec7587c-8c4b-46c3-86d3-a002ec7a5cac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:11.203738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-dec7587c-8c4b-46c3-86d3-a002ec7a5cac None None] MFA Rules not processed for user `fabdcdc25d1d4ef6bc66bd6cadecaaf8`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:11.206662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-dec7587c-8c4b-46c3-86d3-a002ec7a5cac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:11.237534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:11.269299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698703a7-508b-477c-849c-e55a1502a2fc service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:11.279903 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d70b980-09fd-4311-89ae-84a16f76ee76 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:11.280608 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d70b980-09fd-4311-89ae-84a16f76ee76 service swift] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:11.280878 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d70b980-09fd-4311-89ae-84a16f76ee76 service swift] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:11.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d70b980-09fd-4311-89ae-84a16f76ee76 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:11.304643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d70b980-09fd-4311-89ae-84a16f76ee76 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:11.306373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d70b980-09fd-4311-89ae-84a16f76ee76 service swift] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:11.351618 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1138/2268] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:28:11 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 21 01:28:11.384415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1131/2269] 10.222.0.47 () {58 vars in 1018 bytes} [Sun Aug 21 01:28:11 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:28:12.054988 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d70b980-09fd-4311-89ae-84a16f76ee76 service swift] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:12.060194 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9d70b980-09fd-4311-89ae-84a16f76ee76 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:12.163628 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d70b980-09fd-4311-89ae-84a16f76ee76 service swift] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:12.169040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fae1dd2-5694-487b-afcd-d8c7f83150dc service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6365b2328f34ab9b1b73ececbf5cc30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:12.169730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fae1dd2-5694-487b-afcd-d8c7f83150dc service glance] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:12.170088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fae1dd2-5694-487b-afcd-d8c7f83150dc service glance] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:12.170366 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fae1dd2-5694-487b-afcd-d8c7f83150dc service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:12.186894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fae1dd2-5694-487b-afcd-d8c7f83150dc service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:12.188548 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fae1dd2-5694-487b-afcd-d8c7f83150dc service glance] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:12.206094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1139/2270] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:28:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:28:12.658164 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96bda85-91d2-46c2-adcd-8e44862eb2cc None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:12.658582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96bda85-91d2-46c2-adcd-8e44862eb2cc None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:12.658832 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96bda85-91d2-46c2-adcd-8e44862eb2cc None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:12.660741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1132/2271] 10.222.0.47 () {54 vars in 921 bytes} [Sun Aug 21 01:28:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:28:12.669687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb51d303-79af-4746-942c-5bc57410cea2 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:12.670041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb51d303-79af-4746-942c-5bc57410cea2 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:12.670251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb51d303-79af-4746-942c-5bc57410cea2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:12.690844 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780957d7-051d-45d2-8dd1-af0391f50fa8 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:12.691128 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780957d7-051d-45d2-8dd1-af0391f50fa8 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:12.691407 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780957d7-051d-45d2-8dd1-af0391f50fa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:12.712660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-cb51d303-79af-4746-942c-5bc57410cea2 None None] MFA Rules not processed for user `b532ddab1f5e467989edaa9e4b74bd72`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:12.718298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-cb51d303-79af-4746-942c-5bc57410cea2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:12.719367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-780957d7-051d-45d2-8dd1-af0391f50fa8 None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:12.720784 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-780957d7-051d-45d2-8dd1-af0391f50fa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:12.748538 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1133/2273] 10.222.0.47 () {58 vars in 1009 bytes} [Sun Aug 21 01:28:12 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:28:12.754135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1140/2273] 10.222.0.47 () {58 vars in 1018 bytes} [Sun Aug 21 01:28:12 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:12.755121 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780957d7-051d-45d2-8dd1-af0391f50fa8 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:12.765718 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780957d7-051d-45d2-8dd1-af0391f50fa8 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:12.770930 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c327fe39-9ce8-4f7a-9361-3a1b6c9b61c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:12.771436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb51d303-79af-4746-942c-5bc57410cea2 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:12.771618 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c327fe39-9ce8-4f7a-9361-3a1b6c9b61c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:12.771901 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c327fe39-9ce8-4f7a-9361-3a1b6c9b61c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:12.772200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c327fe39-9ce8-4f7a-9361-3a1b6c9b61c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:12.781333 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c327fe39-9ce8-4f7a-9361-3a1b6c9b61c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:12.783105 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c327fe39-9ce8-4f7a-9361-3a1b6c9b61c4 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:12.790882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb51d303-79af-4746-942c-5bc57410cea2 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:12.799306 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1134/2274] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:28:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:28:12.804435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d11c50d-8750-4b15-9f7c-2e467d0e0dff service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:12.805187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d11c50d-8750-4b15-9f7c-2e467d0e0dff service swift] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:12.805582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d11c50d-8750-4b15-9f7c-2e467d0e0dff service swift] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:12.805960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d11c50d-8750-4b15-9f7c-2e467d0e0dff service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:12.815840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d11c50d-8750-4b15-9f7c-2e467d0e0dff service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:12.817680 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d11c50d-8750-4b15-9f7c-2e467d0e0dff service swift] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:12.832011 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1141/2275] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:28:12 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 21 01:28:12.953812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8bc5a8-0fa7-410f-b164-c1fff33aeebf None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:12.957402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8bc5a8-0fa7-410f-b164-c1fff33aeebf None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:12.957402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8bc5a8-0fa7-410f-b164-c1fff33aeebf None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:12.957402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1135/2276] 10.222.0.47 () {54 vars in 921 bytes} [Sun Aug 21 01:28:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:28:12.964996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa39ba03-6b10-40a0-bf0e-6ff86315b953 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:12.965506 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa39ba03-6b10-40a0-bf0e-6ff86315b953 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:12.965988 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa39ba03-6b10-40a0-bf0e-6ff86315b953 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:12.990495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-aa39ba03-6b10-40a0-bf0e-6ff86315b953 None None] MFA Rules not processed for user `fabdcdc25d1d4ef6bc66bd6cadecaaf8`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:12.991738 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-aa39ba03-6b10-40a0-bf0e-6ff86315b953 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:13.049768 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1142/2277] 10.222.0.47 () {58 vars in 1018 bytes} [Sun Aug 21 01:28:12 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:13.062362 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8bc5a8-0fa7-410f-b164-c1fff33aeebf None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:13.088569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8bc5a8-0fa7-410f-b164-c1fff33aeebf None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:13.096590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1608334-79c7-4002-ab07-fcdb8bea8c02 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:13.097271 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1608334-79c7-4002-ab07-fcdb8bea8c02 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:13.097569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1608334-79c7-4002-ab07-fcdb8bea8c02 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:13.097847 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1608334-79c7-4002-ab07-fcdb8bea8c02 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:13.123230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1608334-79c7-4002-ab07-fcdb8bea8c02 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:13.124796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1608334-79c7-4002-ab07-fcdb8bea8c02 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:13.144747 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1136/2278] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:28:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:28:13.974339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313099ba-a20d-4d82-af89-fea698b61d83 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:13.975461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313099ba-a20d-4d82-af89-fea698b61d83 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:13.975901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313099ba-a20d-4d82-af89-fea698b61d83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:14.047711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-313099ba-a20d-4d82-af89-fea698b61d83 None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:14.050545 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-313099ba-a20d-4d82-af89-fea698b61d83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:14.089748 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1143/2279] 10.222.0.47 () {58 vars in 967 bytes} [Sun Aug 21 01:28:13 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:14.145671 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1608334-79c7-4002-ab07-fcdb8bea8c02 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:14.168640 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1608334-79c7-4002-ab07-fcdb8bea8c02 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:14.204903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8e1b94-04f8-47a0-95f6-cec37d8d62bb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:14.205920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8e1b94-04f8-47a0-95f6-cec37d8d62bb service swift] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:14.206058 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8e1b94-04f8-47a0-95f6-cec37d8d62bb service swift] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:14.206317 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8e1b94-04f8-47a0-95f6-cec37d8d62bb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:14.225837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8e1b94-04f8-47a0-95f6-cec37d8d62bb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:14.234264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8e1b94-04f8-47a0-95f6-cec37d8d62bb service swift] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:14.250919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1137/2280] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:28:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 21 01:28:18.570320 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313099ba-a20d-4d82-af89-fea698b61d83 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:18.573224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-313099ba-a20d-4d82-af89-fea698b61d83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:18.597543 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313099ba-a20d-4d82-af89-fea698b61d83 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:18.611901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5613f43-7635-4a80-a256-4dd47a28aa18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:18.612696 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5613f43-7635-4a80-a256-4dd47a28aa18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:18.613069 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5613f43-7635-4a80-a256-4dd47a28aa18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:18.613562 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5613f43-7635-4a80-a256-4dd47a28aa18 admin admin] PATH_INFO: `/v3/users/78a9f62090244baa883ca47efed0ae20` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:18.615579 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5613f43-7635-4a80-a256-4dd47a28aa18 admin admin] RBAC: Authorizing `identity:delete_user(user_id=78a9f62090244baa883ca47efed0ae20)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:18.622879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5613f43-7635-4a80-a256-4dd47a28aa18 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:18.890112 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-c5613f43-7635-4a80-a256-4dd47a28aa18 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '78a9f62090244baa883ca47efed0ae20'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:18.900869 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-c5613f43-7635-4a80-a256-4dd47a28aa18 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '78a9f62090244baa883ca47efed0ae20'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:18.915573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-c5613f43-7635-4a80-a256-4dd47a28aa18 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '78a9f62090244baa883ca47efed0ae20'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:19.012848 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-c5613f43-7635-4a80-a256-4dd47a28aa18 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '78a9f62090244baa883ca47efed0ae20'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:19.013671 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-c5613f43-7635-4a80-a256-4dd47a28aa18 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '78a9f62090244baa883ca47efed0ae20'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:19.022829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1144/2281] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:28:18 2022] DELETE /identity/v3/users/78a9f62090244baa883ca47efed0ae20 => generated 0 bytes in 453 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:28:19.028340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c037cb-b8a9-477b-b51f-d4b657e2b71e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:19.028871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c037cb-b8a9-477b-b51f-d4b657e2b71e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:19.028963 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c037cb-b8a9-477b-b51f-d4b657e2b71e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:19.083079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-57c037cb-b8a9-477b-b51f-d4b657e2b71e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:19.084170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-57c037cb-b8a9-477b-b51f-d4b657e2b71e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:19.090970 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:19.192200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1138/2282] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:28:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:19.199925 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5613f43-7635-4a80-a256-4dd47a28aa18 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:19.201642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c5613f43-7635-4a80-a256-4dd47a28aa18 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:19.245258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:19.296685 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5613f43-7635-4a80-a256-4dd47a28aa18 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:19.301744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b48895-27d7-47a8-943c-ce209d55fb27 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:19.302952 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b48895-27d7-47a8-943c-ce209d55fb27 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:19.303330 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b48895-27d7-47a8-943c-ce209d55fb27 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:19.303690 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b48895-27d7-47a8-943c-ce209d55fb27 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:19.316356 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b48895-27d7-47a8-943c-ce209d55fb27 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:19.317876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b48895-27d7-47a8-943c-ce209d55fb27 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:19.331114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1145/2283] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:28:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:28:19.365473 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c037cb-b8a9-477b-b51f-d4b657e2b71e None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:19.368602 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-57c037cb-b8a9-477b-b51f-d4b657e2b71e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:19.508940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c037cb-b8a9-477b-b51f-d4b657e2b71e None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:19.542492 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf2928d-f811-488f-ab5a-0933276c18f9 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '972b0f1dbfef4d85a6dfc4c5fd839bde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:19.543129 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf2928d-f811-488f-ab5a-0933276c18f9 service placement] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:19.543408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf2928d-f811-488f-ab5a-0933276c18f9 service placement] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:19.543681 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf2928d-f811-488f-ab5a-0933276c18f9 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:19.545835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-faf2928d-f811-488f-ab5a-0933276c18f9 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:19.577809 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf2928d-f811-488f-ab5a-0933276c18f9 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:19.579425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf2928d-f811-488f-ab5a-0933276c18f9 service placement] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:19.597952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1139/2284] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:28:19 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:28:19.756396 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b48895-27d7-47a8-943c-ce209d55fb27 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:19.758102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-43b48895-27d7-47a8-943c-ce209d55fb27 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:19.783034 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b48895-27d7-47a8-943c-ce209d55fb27 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:19.794066 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1aae95-f669-4e44-8a68-043afd39be1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:19.794846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1aae95-f669-4e44-8a68-043afd39be1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:19.795196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1aae95-f669-4e44-8a68-043afd39be1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:19.795461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1aae95-f669-4e44-8a68-043afd39be1d admin admin] PATH_INFO: `/v3/projects/e8775cdf1c9c40e8aac986896c4e0b01` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:19.801442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1aae95-f669-4e44-8a68-043afd39be1d admin admin] RBAC: Authorizing `identity:delete_project(project_id=e8775cdf1c9c40e8aac986896c4e0b01)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:19.801442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1aae95-f669-4e44-8a68-043afd39be1d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:20.003761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-4f1aae95-f669-4e44-8a68-043afd39be1d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e8775cdf1c9c40e8aac986896c4e0b01', 'request_id': 'req-4f1aae95-f669-4e44-8a68-043afd39be1d'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:20.132671 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e3571d-c813-4b1e-ad2f-e8f8fa7c3f90 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:20.133220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e3571d-c813-4b1e-ad2f-e8f8fa7c3f90 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:20.133220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e3571d-c813-4b1e-ad2f-e8f8fa7c3f90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:20.151131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-4f1aae95-f669-4e44-8a68-043afd39be1d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e8775cdf1c9c40e8aac986896c4e0b01', 'request_id': 'req-4f1aae95-f669-4e44-8a68-043afd39be1d'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:20.161502 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-4f1aae95-f669-4e44-8a68-043afd39be1d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e8775cdf1c9c40e8aac986896c4e0b01', 'request_id': 'req-4f1aae95-f669-4e44-8a68-043afd39be1d'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:20.178439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-88e3571d-c813-4b1e-ad2f-e8f8fa7c3f90 None None] MFA Rules not processed for user `03d34d1e66404d259599806d040628ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:20.180266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-88e3571d-c813-4b1e-ad2f-e8f8fa7c3f90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:20.183657 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-4f1aae95-f669-4e44-8a68-043afd39be1d admin admin] The token cache is being invalidate because project e8775cdf1c9c40e8aac986896c4e0b01 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:28:20.184514 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-4f1aae95-f669-4e44-8a68-043afd39be1d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:20.189183 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-4f1aae95-f669-4e44-8a68-043afd39be1d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:20.226751 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1146/2285] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:28:19 2022] DELETE /identity/v3/projects/e8775cdf1c9c40e8aac986896c4e0b01 => generated 0 bytes in 456 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:28:20.413968 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1140/2286] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:28:20 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:28:20.419438 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1aae95-f669-4e44-8a68-043afd39be1d admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:20.421223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-4f1aae95-f669-4e44-8a68-043afd39be1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:20.519925 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1aae95-f669-4e44-8a68-043afd39be1d admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:20.525229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77eb9f37-addf-446c-bb09-06263e3d41bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:20.526468 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eb9f37-addf-446c-bb09-06263e3d41bc admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:20.526732 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eb9f37-addf-446c-bb09-06263e3d41bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:20.527006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eb9f37-addf-446c-bb09-06263e3d41bc admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:20.527789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77eb9f37-addf-446c-bb09-06263e3d41bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:20.529777 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77eb9f37-addf-446c-bb09-06263e3d41bc admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:20.641532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1147/2287] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:28:20 2022] POST /identity/v3/projects => generated 371 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:28:20.645675 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e3571d-c813-4b1e-ad2f-e8f8fa7c3f90 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:20.657011 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e3571d-c813-4b1e-ad2f-e8f8fa7c3f90 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:20.683760 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:20.714467 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aba41a-8dd7-42ea-b636-1cf1f1396f01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:20.715079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aba41a-8dd7-42ea-b636-1cf1f1396f01 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:20.715358 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aba41a-8dd7-42ea-b636-1cf1f1396f01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:20.715627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aba41a-8dd7-42ea-b636-1cf1f1396f01 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:20.716270 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79aba41a-8dd7-42ea-b636-1cf1f1396f01 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:20.718248 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79aba41a-8dd7-42ea-b636-1cf1f1396f01 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:20.805191 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1141/2288] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:28:20 2022] POST /identity/v3/users => generated 292 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:28:20.809495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77eb9f37-addf-446c-bb09-06263e3d41bc admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:20.820942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77eb9f37-addf-446c-bb09-06263e3d41bc admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:20.841679 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1da65ee-2394-4ec0-8247-bb458d99d347 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:20.842308 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1da65ee-2394-4ec0-8247-bb458d99d347 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:20.842585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1da65ee-2394-4ec0-8247-bb458d99d347 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:20.842905 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1da65ee-2394-4ec0-8247-bb458d99d347 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:20.843554 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1da65ee-2394-4ec0-8247-bb458d99d347 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:20.845314 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1da65ee-2394-4ec0-8247-bb458d99d347 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:20.859968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1148/2289] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:28:20 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:28:20.861189 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aba41a-8dd7-42ea-b636-1cf1f1396f01 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:20.875749 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aba41a-8dd7-42ea-b636-1cf1f1396f01 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:20.879131 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aec6d5a-e068-4e2e-8406-059e0dccad63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:20.879899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aec6d5a-e068-4e2e-8406-059e0dccad63 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:20.880266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aec6d5a-e068-4e2e-8406-059e0dccad63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:20.880678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aec6d5a-e068-4e2e-8406-059e0dccad63 admin admin] PATH_INFO: `/v3/projects/df1eeabcc1264062a10d42671ad56c3c/users/8717d3e589424f00abdfcc77395a9db8/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:20.897783 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aec6d5a-e068-4e2e-8406-059e0dccad63 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df1eeabcc1264062a10d42671ad56c3c, user_id=8717d3e589424f00abdfcc77395a9db8, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:20.901327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aec6d5a-e068-4e2e-8406-059e0dccad63 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:20.974228 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1142/2290] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:28:20 2022] PUT /identity/v3/projects/df1eeabcc1264062a10d42671ad56c3c/users/8717d3e589424f00abdfcc77395a9db8/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:28:20.978314 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1da65ee-2394-4ec0-8247-bb458d99d347 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:20.988763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1da65ee-2394-4ec0-8247-bb458d99d347 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:21.013815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:21.050100 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b6fdba-44c5-4e3e-ba16-9e589fa8866a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:21.050742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b6fdba-44c5-4e3e-ba16-9e589fa8866a admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:21.051074 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b6fdba-44c5-4e3e-ba16-9e589fa8866a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:21.051677 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b6fdba-44c5-4e3e-ba16-9e589fa8866a admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:21.052357 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b6fdba-44c5-4e3e-ba16-9e589fa8866a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:21.054041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b6fdba-44c5-4e3e-ba16-9e589fa8866a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:21.065574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1149/2291] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:28:20 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:28:21.069668 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aec6d5a-e068-4e2e-8406-059e0dccad63 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:21.081556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aec6d5a-e068-4e2e-8406-059e0dccad63 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:21.086453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c43bce-cdf2-47ca-b625-51216594ea41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:21.087113 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c43bce-cdf2-47ca-b625-51216594ea41 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:21.087390 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c43bce-cdf2-47ca-b625-51216594ea41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:21.087661 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c43bce-cdf2-47ca-b625-51216594ea41 admin admin] PATH_INFO: `/v3/projects/df1eeabcc1264062a10d42671ad56c3c/users/8717d3e589424f00abdfcc77395a9db8/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:21.089674 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c43bce-cdf2-47ca-b625-51216594ea41 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df1eeabcc1264062a10d42671ad56c3c, user_id=8717d3e589424f00abdfcc77395a9db8, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:21.092494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c43bce-cdf2-47ca-b625-51216594ea41 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:21.103559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1143/2292] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:28:21 2022] PUT /identity/v3/projects/df1eeabcc1264062a10d42671ad56c3c/users/8717d3e589424f00abdfcc77395a9db8/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:28:24.101574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c17eb4a-53e8-492e-a0d4-f1aa83f52b19 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:24.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c17eb4a-53e8-492e-a0d4-f1aa83f52b19 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:24.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c17eb4a-53e8-492e-a0d4-f1aa83f52b19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:24.134114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-7c17eb4a-53e8-492e-a0d4-f1aa83f52b19 None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:24.135830 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-7c17eb4a-53e8-492e-a0d4-f1aa83f52b19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:24.217192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1150/2293] 10.222.0.47 () {58 vars in 1009 bytes} [Sun Aug 21 01:28:24 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:24.236861 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c43bce-cdf2-47ca-b625-51216594ea41 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:24.236861 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-41c43bce-cdf2-47ca-b625-51216594ea41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:24.310308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c43bce-cdf2-47ca-b625-51216594ea41 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:24.315756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edb539c-b8ae-4b60-97e1-fc2524561288 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:24.316428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edb539c-b8ae-4b60-97e1-fc2524561288 service swift] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:24.316705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edb539c-b8ae-4b60-97e1-fc2524561288 service swift] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:24.316980 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edb539c-b8ae-4b60-97e1-fc2524561288 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:24.328145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edb539c-b8ae-4b60-97e1-fc2524561288 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:24.333450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edb539c-b8ae-4b60-97e1-fc2524561288 service swift] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:24.354955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1144/2294] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:28:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 21 01:28:26.051056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c17eb4a-53e8-492e-a0d4-f1aa83f52b19 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:26.052612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-7c17eb4a-53e8-492e-a0d4-f1aa83f52b19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:26.171944 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c17eb4a-53e8-492e-a0d4-f1aa83f52b19 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:26.177382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d737a134-6b15-4194-af3d-a61ab42be44e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:26.178116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d737a134-6b15-4194-af3d-a61ab42be44e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:26.178689 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d737a134-6b15-4194-af3d-a61ab42be44e admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:26.179219 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d737a134-6b15-4194-af3d-a61ab42be44e admin admin] PATH_INFO: `/v3/users/c7359551e76d42849f51af363fcd4386` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:26.181435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d737a134-6b15-4194-af3d-a61ab42be44e admin admin] RBAC: Authorizing `identity:delete_user(user_id=c7359551e76d42849f51af363fcd4386)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:26.187575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d737a134-6b15-4194-af3d-a61ab42be44e admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:26.410361 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d737a134-6b15-4194-af3d-a61ab42be44e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c7359551e76d42849f51af363fcd4386'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:26.433211 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d737a134-6b15-4194-af3d-a61ab42be44e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c7359551e76d42849f51af363fcd4386'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:26.456048 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:26.461797 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d737a134-6b15-4194-af3d-a61ab42be44e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c7359551e76d42849f51af363fcd4386'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:26.537116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d737a134-6b15-4194-af3d-a61ab42be44e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c7359551e76d42849f51af363fcd4386'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:26.537817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d737a134-6b15-4194-af3d-a61ab42be44e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c7359551e76d42849f51af363fcd4386'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:26.548067 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1151/2295] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:28:26 2022] DELETE /identity/v3/users/c7359551e76d42849f51af363fcd4386 => generated 0 bytes in 497 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:28:26.552606 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b5b932-834d-4408-94ce-4677d9bed7dc None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:26.553503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b5b932-834d-4408-94ce-4677d9bed7dc None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:26.553933 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b5b932-834d-4408-94ce-4677d9bed7dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:26.590072 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-86b5b932-834d-4408-94ce-4677d9bed7dc None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:26.591655 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-86b5b932-834d-4408-94ce-4677d9bed7dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:26.698878 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1145/2296] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:28:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:26.772991 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d737a134-6b15-4194-af3d-a61ab42be44e admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:26.774524 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-d737a134-6b15-4194-af3d-a61ab42be44e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:26.981510 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d737a134-6b15-4194-af3d-a61ab42be44e admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:26.998029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f99648-ca83-44ff-a093-b3513d5ded98 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:26.998678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f99648-ca83-44ff-a093-b3513d5ded98 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:26.998988 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f99648-ca83-44ff-a093-b3513d5ded98 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:26.999200 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f99648-ca83-44ff-a093-b3513d5ded98 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:27.038717 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f99648-ca83-44ff-a093-b3513d5ded98 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:27.040182 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f99648-ca83-44ff-a093-b3513d5ded98 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:27.068609 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1152/2297] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:28:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:28:28.746013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b5b932-834d-4408-94ce-4677d9bed7dc None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:28.747770 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-86b5b932-834d-4408-94ce-4677d9bed7dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:28.763498 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b5b932-834d-4408-94ce-4677d9bed7dc None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:28.775822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34494edc-585d-4152-8645-fe7bfde936af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:28.777271 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34494edc-585d-4152-8645-fe7bfde936af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:28.780714 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34494edc-585d-4152-8645-fe7bfde936af admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:28.780714 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34494edc-585d-4152-8645-fe7bfde936af admin admin] PATH_INFO: `/v3/projects/dfa1623618fe43748f6a3efb03373892` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:28.780714 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34494edc-585d-4152-8645-fe7bfde936af admin admin] RBAC: Authorizing `identity:delete_project(project_id=dfa1623618fe43748f6a3efb03373892)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:28.783678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34494edc-585d-4152-8645-fe7bfde936af admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:28.799311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:28.951259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-34494edc-585d-4152-8645-fe7bfde936af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dfa1623618fe43748f6a3efb03373892', 'request_id': 'req-34494edc-585d-4152-8645-fe7bfde936af'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:28.957821 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-34494edc-585d-4152-8645-fe7bfde936af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dfa1623618fe43748f6a3efb03373892', 'request_id': 'req-34494edc-585d-4152-8645-fe7bfde936af'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:28.972246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-34494edc-585d-4152-8645-fe7bfde936af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dfa1623618fe43748f6a3efb03373892', 'request_id': 'req-34494edc-585d-4152-8645-fe7bfde936af'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:29.240669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-34494edc-585d-4152-8645-fe7bfde936af admin admin] The token cache is being invalidate because project dfa1623618fe43748f6a3efb03373892 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:28:29.241083 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-34494edc-585d-4152-8645-fe7bfde936af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:29.242410 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-34494edc-585d-4152-8645-fe7bfde936af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:28:29.244362 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1146/2298] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:28:28 2022] DELETE /identity/v3/projects/dfa1623618fe43748f6a3efb03373892 => generated 0 bytes in 499 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:28:29.249030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f195c5-7edc-46bc-8a95-32ddd1034c1c None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:29.254187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f195c5-7edc-46bc-8a95-32ddd1034c1c None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:29.254187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f195c5-7edc-46bc-8a95-32ddd1034c1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:29.284599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-90f195c5-7edc-46bc-8a95-32ddd1034c1c None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:29.285928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-90f195c5-7edc-46bc-8a95-32ddd1034c1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:29.439781 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1153/2299] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:28:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:29.447000 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01d12c5-15e1-476a-b815-cd807a375c0d None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:29.447712 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01d12c5-15e1-476a-b815-cd807a375c0d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:29.448033 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01d12c5-15e1-476a-b815-cd807a375c0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:29.480835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-d01d12c5-15e1-476a-b815-cd807a375c0d None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:29.482697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d01d12c5-15e1-476a-b815-cd807a375c0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:29.494811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1147/2300] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:28:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:28:29.498694 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f195c5-7edc-46bc-8a95-32ddd1034c1c None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:29.511716 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f195c5-7edc-46bc-8a95-32ddd1034c1c None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:29.524586 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04537c5-499e-43dd-b3f2-0190f3bf7466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:29.532305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04537c5-499e-43dd-b3f2-0190f3bf7466 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:29.532305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04537c5-499e-43dd-b3f2-0190f3bf7466 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:29.532305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04537c5-499e-43dd-b3f2-0190f3bf7466 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:29.532305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04537c5-499e-43dd-b3f2-0190f3bf7466 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:28:29.532305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04537c5-499e-43dd-b3f2-0190f3bf7466 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:29.532305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04537c5-499e-43dd-b3f2-0190f3bf7466 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:29.553678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1154/2301] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:28:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:28:29.554774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01d12c5-15e1-476a-b815-cd807a375c0d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:29.568697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01d12c5-15e1-476a-b815-cd807a375c0d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:29.583934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1e5792-1416-402c-bb78-f0cdadd59a61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:29.583934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1e5792-1416-402c-bb78-f0cdadd59a61 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:29.583934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1e5792-1416-402c-bb78-f0cdadd59a61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:29.583934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1e5792-1416-402c-bb78-f0cdadd59a61 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:29.585939 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1e5792-1416-402c-bb78-f0cdadd59a61 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:29.592006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1e5792-1416-402c-bb78-f0cdadd59a61 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:29.766018 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1148/2302] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:28:29 2022] POST /identity/v3/projects => generated 383 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:28:29.774885 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04537c5-499e-43dd-b3f2-0190f3bf7466 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:29.783592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04537c5-499e-43dd-b3f2-0190f3bf7466 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:29.941622 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afe3154-8276-4321-9fa3-eb8802405f71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:29.943252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afe3154-8276-4321-9fa3-eb8802405f71 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:29.944515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afe3154-8276-4321-9fa3-eb8802405f71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:29.945898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afe3154-8276-4321-9fa3-eb8802405f71 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:29.949413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4afe3154-8276-4321-9fa3-eb8802405f71 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:29.959605 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4afe3154-8276-4321-9fa3-eb8802405f71 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:30.081197 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1e5792-1416-402c-bb78-f0cdadd59a61 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:30.092379 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1155/2303] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:28:29 2022] POST /identity/v3/users => generated 292 bytes in 323 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:28:30.104805 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1e5792-1416-402c-bb78-f0cdadd59a61 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:30.115285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85263ab-2c2c-4a29-804e-de5d04a0c191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:30.115917 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85263ab-2c2c-4a29-804e-de5d04a0c191 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:30.116315 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85263ab-2c2c-4a29-804e-de5d04a0c191 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:30.116592 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85263ab-2c2c-4a29-804e-de5d04a0c191 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:30.117241 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85263ab-2c2c-4a29-804e-de5d04a0c191 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:30.118851 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85263ab-2c2c-4a29-804e-de5d04a0c191 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:30.142840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afe3154-8276-4321-9fa3-eb8802405f71 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:30.143947 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1149/2304] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:28:30 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:28:30.156593 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afe3154-8276-4321-9fa3-eb8802405f71 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:30.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9864d368-be2d-4ca7-93c6-e83172df1923 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:30.173816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9864d368-be2d-4ca7-93c6-e83172df1923 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:30.174163 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9864d368-be2d-4ca7-93c6-e83172df1923 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:30.174567 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75547d7-5c2e-4079-a072-7edd6ec8223d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:30.175452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75547d7-5c2e-4079-a072-7edd6ec8223d admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:30.175745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75547d7-5c2e-4079-a072-7edd6ec8223d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:30.176021 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75547d7-5c2e-4079-a072-7edd6ec8223d admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:30.176724 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75547d7-5c2e-4079-a072-7edd6ec8223d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:30.180774 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75547d7-5c2e-4079-a072-7edd6ec8223d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:30.183144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:DELETE=2,keystone:SELECT=94,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:30.205561 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1156/2305] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:28:30 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:28:30.211437 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75547d7-5c2e-4079-a072-7edd6ec8223d admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:30.249533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75547d7-5c2e-4079-a072-7edd6ec8223d admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:30.252553 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-9864d368-be2d-4ca7-93c6-e83172df1923 None None] MFA Rules not processed for user `8717d3e589424f00abdfcc77395a9db8`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:30.253809 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9864d368-be2d-4ca7-93c6-e83172df1923 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:30.264847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820fcabc-87a8-4cb5-bef0-92e82de205e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:30.265540 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820fcabc-87a8-4cb5-bef0-92e82de205e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:30.265818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820fcabc-87a8-4cb5-bef0-92e82de205e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:30.266093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820fcabc-87a8-4cb5-bef0-92e82de205e2 admin admin] PATH_INFO: `/v3/projects/67f492f96d314cfea70bdbfa3caaee9e/users/2325a3d547d043d2a9ac95986187027b/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:30.304397 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820fcabc-87a8-4cb5-bef0-92e82de205e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=67f492f96d314cfea70bdbfa3caaee9e, user_id=2325a3d547d043d2a9ac95986187027b, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:30.307499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820fcabc-87a8-4cb5-bef0-92e82de205e2 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:30.401342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1150/2306] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:28:30 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:28:30.424799 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9864d368-be2d-4ca7-93c6-e83172df1923 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:30.424799 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9864d368-be2d-4ca7-93c6-e83172df1923 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:30.450411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1157/2307] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:28:30 2022] PUT /identity/v3/projects/67f492f96d314cfea70bdbfa3caaee9e/users/2325a3d547d043d2a9ac95986187027b/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:28:30.456191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021def61-c00f-4b17-b617-75d6c24dfd7b None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:30.456563 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021def61-c00f-4b17-b617-75d6c24dfd7b None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:30.456738 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021def61-c00f-4b17-b617-75d6c24dfd7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:30.509171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-021def61-c00f-4b17-b617-75d6c24dfd7b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:30.510431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-021def61-c00f-4b17-b617-75d6c24dfd7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:30.535021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:30.612366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9864d368-be2d-4ca7-93c6-e83172df1923 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:30.678586 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1158/2308] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:28:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:30.692593 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021def61-c00f-4b17-b617-75d6c24dfd7b None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:30.694077 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-021def61-c00f-4b17-b617-75d6c24dfd7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:30.785193 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021def61-c00f-4b17-b617-75d6c24dfd7b None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:30.789243 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac928798-7f3a-4b74-be46-35e71e3fac59 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:30.789471 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e537838f-608a-45b2-a31f-8be9c293f4bd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:30.790016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac928798-7f3a-4b74-be46-35e71e3fac59 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:30.790298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac928798-7f3a-4b74-be46-35e71e3fac59 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:30.790570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac928798-7f3a-4b74-be46-35e71e3fac59 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:30.790755 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e537838f-608a-45b2-a31f-8be9c293f4bd service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:30.791201 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e537838f-608a-45b2-a31f-8be9c293f4bd service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:30.791467 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e537838f-608a-45b2-a31f-8be9c293f4bd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:30.799623 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e537838f-608a-45b2-a31f-8be9c293f4bd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:30.812335 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac928798-7f3a-4b74-be46-35e71e3fac59 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:30.813672 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac928798-7f3a-4b74-be46-35e71e3fac59 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:30.841307 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1159/2309] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:28:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:28:30.879245 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e537838f-608a-45b2-a31f-8be9c293f4bd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:30.885388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e537838f-608a-45b2-a31f-8be9c293f4bd service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:30.896404 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1151/2310] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:28:30 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 481 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:28:34.024552 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156959de-a36a-482a-85af-bc7437e95262 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:34.028302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156959de-a36a-482a-85af-bc7437e95262 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:34.028302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156959de-a36a-482a-85af-bc7437e95262 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:34.136808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-156959de-a36a-482a-85af-bc7437e95262 None None] MFA Rules not processed for user `2325a3d547d043d2a9ac95986187027b`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:34.139203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-156959de-a36a-482a-85af-bc7437e95262 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:34.236142 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1160/2311] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:28:34 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:34.248973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e537838f-608a-45b2-a31f-8be9c293f4bd service nova] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:34.251825 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e537838f-608a-45b2-a31f-8be9c293f4bd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:34.361477 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e537838f-608a-45b2-a31f-8be9c293f4bd service nova] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:34.375662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d68e7e-3ec8-4276-ac4d-ef982788ccee service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:34.377467 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d68e7e-3ec8-4276-ac4d-ef982788ccee service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:34.378014 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d68e7e-3ec8-4276-ac4d-ef982788ccee service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:34.378661 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d68e7e-3ec8-4276-ac4d-ef982788ccee service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:34.391746 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d68e7e-3ec8-4276-ac4d-ef982788ccee service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:34.395081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d68e7e-3ec8-4276-ac4d-ef982788ccee service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:34.426090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1152/2312] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:28:34 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:28:35.708350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfefde2e-8334-4e87-a636-6620ace175bf None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:35.709103 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfefde2e-8334-4e87-a636-6620ace175bf None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:35.709103 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfefde2e-8334-4e87-a636-6620ace175bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:35.717411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:35.729692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-bfefde2e-8334-4e87-a636-6620ace175bf None None] MFA Rules not processed for user `03d34d1e66404d259599806d040628ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:35.731207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-bfefde2e-8334-4e87-a636-6620ace175bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:35.738831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1161/2313] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:28:35 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:28:36.554791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7933a969-fe1d-4a24-8ed8-023795b503b4 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:36.555568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7933a969-fe1d-4a24-8ed8-023795b503b4 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:36.555568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7933a969-fe1d-4a24-8ed8-023795b503b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:36.575864 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-7933a969-fe1d-4a24-8ed8-023795b503b4 None None] MFA Rules not processed for user `2325a3d547d043d2a9ac95986187027b`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:36.577089 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7933a969-fe1d-4a24-8ed8-023795b503b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:36.584207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1153/2314] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:28:36 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:28:45.827118 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:46.570403 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:47.502961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfefde2e-8334-4e87-a636-6620ace175bf None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:47.520577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfefde2e-8334-4e87-a636-6620ace175bf None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:47.527409 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12df93aa-6a47-4913-8b31-03cb89120b2a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:47.529098 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12df93aa-6a47-4913-8b31-03cb89120b2a service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:47.530006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12df93aa-6a47-4913-8b31-03cb89120b2a service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:47.530734 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12df93aa-6a47-4913-8b31-03cb89120b2a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:47.533968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-12df93aa-6a47-4913-8b31-03cb89120b2a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:47.648540 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12df93aa-6a47-4913-8b31-03cb89120b2a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:47.650000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12df93aa-6a47-4913-8b31-03cb89120b2a service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:47.662925 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1162/2315] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:28:47 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:28:49.647428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1341f8f7-78ef-4dea-9e03-28359fdce81b None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:49.648970 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1341f8f7-78ef-4dea-9e03-28359fdce81b None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:49.648970 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1341f8f7-78ef-4dea-9e03-28359fdce81b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:49.681451 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-1341f8f7-78ef-4dea-9e03-28359fdce81b None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:28:49.684501 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-1341f8f7-78ef-4dea-9e03-28359fdce81b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:49.781704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1154/2316] 10.222.0.47 () {58 vars in 967 bytes} [Sun Aug 21 01:28:49 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:28:49.806501 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12df93aa-6a47-4913-8b31-03cb89120b2a service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:28:49.808341 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-12df93aa-6a47-4913-8b31-03cb89120b2a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:28:49.902937 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12df93aa-6a47-4913-8b31-03cb89120b2a service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:28:49.908569 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7487b067-df6e-478e-8cce-abf1aba860d4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:28:49.908958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7487b067-df6e-478e-8cce-abf1aba860d4 service swift] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:28:49.909234 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7487b067-df6e-478e-8cce-abf1aba860d4 service swift] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:28:49.909508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7487b067-df6e-478e-8cce-abf1aba860d4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:28:49.918906 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7487b067-df6e-478e-8cce-abf1aba860d4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:28:49.922626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7487b067-df6e-478e-8cce-abf1aba860d4 service swift] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:28:49.939784 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1163/2317] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:28:49 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 21 01:28:59.765639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=25 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:28:59.931985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=62 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:29:21.335571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1341f8f7-78ef-4dea-9e03-28359fdce81b None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:29:21.350663 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1341f8f7-78ef-4dea-9e03-28359fdce81b None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:29:21.355625 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e678955-16e8-46d9-8559-d23a80a8147d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:29:21.356291 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e678955-16e8-46d9-8559-d23a80a8147d service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:29:21.356570 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e678955-16e8-46d9-8559-d23a80a8147d service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:29:21.356837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e678955-16e8-46d9-8559-d23a80a8147d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:29:21.426229 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e678955-16e8-46d9-8559-d23a80a8147d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:29:21.428790 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e678955-16e8-46d9-8559-d23a80a8147d service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:29:21.443132 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1155/2318] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:29:21 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:29:22.889465 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbbfdb0-3d93-4081-a50d-5a1855292052 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:29:22.891238 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbbfdb0-3d93-4081-a50d-5a1855292052 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:29:22.891238 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbbfdb0-3d93-4081-a50d-5a1855292052 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:29:22.912538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-7bbbfdb0-3d93-4081-a50d-5a1855292052 None None] MFA Rules not processed for user `d84b6d05decb47c7b55ac782d5839c1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:29:22.913779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-7bbbfdb0-3d93-4081-a50d-5a1855292052 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:29:22.983845 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1164/2319] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:29:22 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:29:31.437301 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=30 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:29:32.973220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=25 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:29:59.941737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e678955-16e8-46d9-8559-d23a80a8147d service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:29:59.950772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e678955-16e8-46d9-8559-d23a80a8147d service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:29:59.954566 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7d4e20-da3d-4483-aacc-1623b94e73b4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:29:59.955185 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7d4e20-da3d-4483-aacc-1623b94e73b4 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:29:59.955458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7d4e20-da3d-4483-aacc-1623b94e73b4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:29:59.955718 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7d4e20-da3d-4483-aacc-1623b94e73b4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:29:59.957453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-4e7d4e20-da3d-4483-aacc-1623b94e73b4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:29:59.971277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7d4e20-da3d-4483-aacc-1623b94e73b4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:29:59.972807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7d4e20-da3d-4483-aacc-1623b94e73b4 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:29:59.984381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1156/2320] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:29:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:30:00.677536 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbbfdb0-3d93-4081-a50d-5a1855292052 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:30:00.687859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbbfdb0-3d93-4081-a50d-5a1855292052 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:30:00.691440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5787fe4a-176e-45fe-a9ae-8e5cad03939a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:30:00.692039 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5787fe4a-176e-45fe-a9ae-8e5cad03939a service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:30:00.692345 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5787fe4a-176e-45fe-a9ae-8e5cad03939a service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:30:00.692642 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5787fe4a-176e-45fe-a9ae-8e5cad03939a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:30:00.694624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-5787fe4a-176e-45fe-a9ae-8e5cad03939a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:30:00.708683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5787fe4a-176e-45fe-a9ae-8e5cad03939a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:30:00.710100 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5787fe4a-176e-45fe-a9ae-8e5cad03939a service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:30:00.721085 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1165/2321] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:30:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:30:01.672924 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1ac585-f608-4f57-a80d-db22acd27bad None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:30:01.673454 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1ac585-f608-4f57-a80d-db22acd27bad None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:30:01.673454 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1ac585-f608-4f57-a80d-db22acd27bad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:30:01.692712 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-dc1ac585-f608-4f57-a80d-db22acd27bad None None] MFA Rules not processed for user `2325a3d547d043d2a9ac95986187027b`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:30:01.694250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-dc1ac585-f608-4f57-a80d-db22acd27bad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:30:01.701943 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1157/2322] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:30:01 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:30:10.716163 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=6 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:30:10.957485 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667cbe28-bc5c-4451-a15d-66ca5fd99d59 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:30:10.957800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667cbe28-bc5c-4451-a15d-66ca5fd99d59 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:30:10.963108 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667cbe28-bc5c-4451-a15d-66ca5fd99d59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:30:10.980199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-667cbe28-bc5c-4451-a15d-66ca5fd99d59 None None] MFA Rules not processed for user `0f4a35c46e4741c1b238f24d203b416d`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:30:10.981423 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-667cbe28-bc5c-4451-a15d-66ca5fd99d59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:30:11.044167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1166/2323] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:30:10 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:30:11.687406 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=9 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:30:15.492737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1ac585-f608-4f57-a80d-db22acd27bad None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:30:15.538500 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1ac585-f608-4f57-a80d-db22acd27bad None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:30:15.546620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2f5dbb-a35c-4648-bb4b-9a19ef68fb42 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:30:15.547560 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2f5dbb-a35c-4648-bb4b-9a19ef68fb42 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:30:15.547903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2f5dbb-a35c-4648-bb4b-9a19ef68fb42 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:30:15.548506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2f5dbb-a35c-4648-bb4b-9a19ef68fb42 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:30:15.551071 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-be2f5dbb-a35c-4648-bb4b-9a19ef68fb42 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:30:15.630038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2f5dbb-a35c-4648-bb4b-9a19ef68fb42 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:30:15.632432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2f5dbb-a35c-4648-bb4b-9a19ef68fb42 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:30:15.643440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1158/2324] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:30:15 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:30:17.107749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667cbe28-bc5c-4451-a15d-66ca5fd99d59 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:30:17.126776 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667cbe28-bc5c-4451-a15d-66ca5fd99d59 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:30:17.130750 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8262ed-de59-4bdc-8d2e-7615cf1c4ad3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:30:17.131224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8262ed-de59-4bdc-8d2e-7615cf1c4ad3 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:30:17.131495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8262ed-de59-4bdc-8d2e-7615cf1c4ad3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:30:17.131762 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8262ed-de59-4bdc-8d2e-7615cf1c4ad3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:30:17.133756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3d8262ed-de59-4bdc-8d2e-7615cf1c4ad3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:30:17.148090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8262ed-de59-4bdc-8d2e-7615cf1c4ad3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:30:17.149602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8262ed-de59-4bdc-8d2e-7615cf1c4ad3 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:30:17.160366 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1167/2325] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:30:17 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:30:18.676848 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2f5dbb-a35c-4648-bb4b-9a19ef68fb42 service nova] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:30:18.688400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2f5dbb-a35c-4648-bb4b-9a19ef68fb42 service nova] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:30:18.695067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639af5bc-7c85-4af0-9fa9-9cdf9d267354 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:30:18.696385 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639af5bc-7c85-4af0-9fa9-9cdf9d267354 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:30:18.696385 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639af5bc-7c85-4af0-9fa9-9cdf9d267354 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:30:18.696385 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639af5bc-7c85-4af0-9fa9-9cdf9d267354 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:30:18.704228 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-639af5bc-7c85-4af0-9fa9-9cdf9d267354 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:30:18.716514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639af5bc-7c85-4af0-9fa9-9cdf9d267354 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:30:18.718749 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639af5bc-7c85-4af0-9fa9-9cdf9d267354 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:30:18.730196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1159/2326] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:30:18 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:30:21.114008 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578b82ea-90fe-4fc1-b374-dfab1ceacb98 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:30:21.114504 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578b82ea-90fe-4fc1-b374-dfab1ceacb98 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:30:21.114602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578b82ea-90fe-4fc1-b374-dfab1ceacb98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:30:21.135440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-578b82ea-90fe-4fc1-b374-dfab1ceacb98 None None] MFA Rules not processed for user `0cd8d822f15943a78575c3744692cdb9`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:30:21.136688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-578b82ea-90fe-4fc1-b374-dfab1ceacb98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:30:21.146215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1168/2327] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:30:21 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:30:21.473984 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639af5bc-7c85-4af0-9fa9-9cdf9d267354 service nova] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:30:21.485912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639af5bc-7c85-4af0-9fa9-9cdf9d267354 service nova] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:30:21.489633 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637e3a65-c3fc-4308-b0fd-cced9b1b3a64 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:30:21.490243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637e3a65-c3fc-4308-b0fd-cced9b1b3a64 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:30:21.490526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637e3a65-c3fc-4308-b0fd-cced9b1b3a64 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:30:21.490794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637e3a65-c3fc-4308-b0fd-cced9b1b3a64 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:30:21.492576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-637e3a65-c3fc-4308-b0fd-cced9b1b3a64 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:30:21.503269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637e3a65-c3fc-4308-b0fd-cced9b1b3a64 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:30:21.504877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637e3a65-c3fc-4308-b0fd-cced9b1b3a64 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:30:21.514363 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1160/2328] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:30:21 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:30:22.041541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578b82ea-90fe-4fc1-b374-dfab1ceacb98 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:30:22.043000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-578b82ea-90fe-4fc1-b374-dfab1ceacb98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:30:22.127050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578b82ea-90fe-4fc1-b374-dfab1ceacb98 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:30:22.132702 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd713bcb-8bcc-4caa-84b3-08368dcb28f0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '972b0f1dbfef4d85a6dfc4c5fd839bde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:30:22.133404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd713bcb-8bcc-4caa-84b3-08368dcb28f0 service placement] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:30:22.133836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd713bcb-8bcc-4caa-84b3-08368dcb28f0 service placement] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:30:22.134203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd713bcb-8bcc-4caa-84b3-08368dcb28f0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:30:22.136310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-bd713bcb-8bcc-4caa-84b3-08368dcb28f0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:30:22.149057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd713bcb-8bcc-4caa-84b3-08368dcb28f0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:30:22.150733 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd713bcb-8bcc-4caa-84b3-08368dcb28f0 service placement] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:30:22.161520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1169/2329] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:30:22 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:30:30.059869 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637e3a65-c3fc-4308-b0fd-cced9b1b3a64 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:30:30.061340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-637e3a65-c3fc-4308-b0fd-cced9b1b3a64 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:30:30.072336 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637e3a65-c3fc-4308-b0fd-cced9b1b3a64 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:30:30.076312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89456377-eb19-436f-8fe2-9abcda99caa2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:30:30.076998 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89456377-eb19-436f-8fe2-9abcda99caa2 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:30:30.077281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89456377-eb19-436f-8fe2-9abcda99caa2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:30:30.077550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89456377-eb19-436f-8fe2-9abcda99caa2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:30:30.079538 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-89456377-eb19-436f-8fe2-9abcda99caa2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:30:30.094930 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89456377-eb19-436f-8fe2-9abcda99caa2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:30:30.094930 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89456377-eb19-436f-8fe2-9abcda99caa2 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:30:30.104958 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1161/2330] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:30:30 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:30:32.157130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=66 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:30:40.099941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=46 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:30:45.138524 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd713bcb-8bcc-4caa-84b3-08368dcb28f0 service placement] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:30:45.141922 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-bd713bcb-8bcc-4caa-84b3-08368dcb28f0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:30:45.176661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd713bcb-8bcc-4caa-84b3-08368dcb28f0 service placement] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:30:45.185434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9bb8de-66c8-4989-9479-0643fc9d8814 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:30:45.186431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9bb8de-66c8-4989-9479-0643fc9d8814 service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:30:45.186776 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9bb8de-66c8-4989-9479-0643fc9d8814 service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:30:45.187274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9bb8de-66c8-4989-9479-0643fc9d8814 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:30:45.189272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-af9bb8de-66c8-4989-9479-0643fc9d8814 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:30:45.203972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9bb8de-66c8-4989-9479-0643fc9d8814 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:30:45.205920 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9bb8de-66c8-4989-9479-0643fc9d8814 service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:30:45.221124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1170/2331] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:30:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:30:53.291761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89456377-eb19-436f-8fe2-9abcda99caa2 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:30:53.309801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89456377-eb19-436f-8fe2-9abcda99caa2 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:30:53.317364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77e3ff0-5ac1-4ce8-90ed-abb2f7d2447b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:30:53.323356 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77e3ff0-5ac1-4ce8-90ed-abb2f7d2447b service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:30:53.323356 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77e3ff0-5ac1-4ce8-90ed-abb2f7d2447b service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:30:53.323356 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77e3ff0-5ac1-4ce8-90ed-abb2f7d2447b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:30:53.323356 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d77e3ff0-5ac1-4ce8-90ed-abb2f7d2447b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:30:53.339634 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77e3ff0-5ac1-4ce8-90ed-abb2f7d2447b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:30:53.342698 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77e3ff0-5ac1-4ce8-90ed-abb2f7d2447b service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:30:53.365141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1162/2332] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:30:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:30:55.213142 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=6 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:30:55.483484 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9bb8de-66c8-4989-9479-0643fc9d8814 service nova] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:30:55.493244 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9bb8de-66c8-4989-9479-0643fc9d8814 service nova] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:30:55.496968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c8fb51-84c3-481a-8343-b8c9ed962565 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:30:55.497524 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c8fb51-84c3-481a-8343-b8c9ed962565 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:30:55.497798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c8fb51-84c3-481a-8343-b8c9ed962565 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:30:55.498066 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c8fb51-84c3-481a-8343-b8c9ed962565 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:30:55.499780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-48c8fb51-84c3-481a-8343-b8c9ed962565 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:30:55.514133 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c8fb51-84c3-481a-8343-b8c9ed962565 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:30:55.515314 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c8fb51-84c3-481a-8343-b8c9ed962565 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:30:55.525923 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1171/2333] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:30:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:31:00.880142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77e3ff0-5ac1-4ce8-90ed-abb2f7d2447b service nova] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:00.880873 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d77e3ff0-5ac1-4ce8-90ed-abb2f7d2447b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:00.895496 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77e3ff0-5ac1-4ce8-90ed-abb2f7d2447b service nova] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:00.900352 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695c4680-3e12-4a62-8b41-db9c4e8f8802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:00.901054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695c4680-3e12-4a62-8b41-db9c4e8f8802 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:00.901335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695c4680-3e12-4a62-8b41-db9c4e8f8802 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:00.903961 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695c4680-3e12-4a62-8b41-db9c4e8f8802 admin admin] PATH_INFO: `/v3/users/d84b6d05decb47c7b55ac782d5839c1a` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:00.903961 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695c4680-3e12-4a62-8b41-db9c4e8f8802 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d84b6d05decb47c7b55ac782d5839c1a)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:00.904952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695c4680-3e12-4a62-8b41-db9c4e8f8802 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:01.221961 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-695c4680-3e12-4a62-8b41-db9c4e8f8802 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd84b6d05decb47c7b55ac782d5839c1a'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:01.229312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-695c4680-3e12-4a62-8b41-db9c4e8f8802 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd84b6d05decb47c7b55ac782d5839c1a'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:01.230061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-695c4680-3e12-4a62-8b41-db9c4e8f8802 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd84b6d05decb47c7b55ac782d5839c1a'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:01.243508 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-695c4680-3e12-4a62-8b41-db9c4e8f8802 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd84b6d05decb47c7b55ac782d5839c1a'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:01.380674 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-695c4680-3e12-4a62-8b41-db9c4e8f8802 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd84b6d05decb47c7b55ac782d5839c1a'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:01.393142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1163/2334] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:31:00 2022] DELETE /identity/v3/users/d84b6d05decb47c7b55ac782d5839c1a => generated 0 bytes in 510 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:31:01.689414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c8fb51-84c3-481a-8343-b8c9ed962565 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:01.700807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c8fb51-84c3-481a-8343-b8c9ed962565 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:01.766325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6904aef-173d-42f6-a2de-c75e4f31f5d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:01.766972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6904aef-173d-42f6-a2de-c75e4f31f5d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:01.767256 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6904aef-173d-42f6-a2de-c75e4f31f5d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:01.767526 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6904aef-173d-42f6-a2de-c75e4f31f5d8 admin admin] PATH_INFO: `/v3/projects/9774c354b8b048768f2071acfb48a4bf` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:01.768719 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6904aef-173d-42f6-a2de-c75e4f31f5d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9774c354b8b048768f2071acfb48a4bf)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:01.770738 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6904aef-173d-42f6-a2de-c75e4f31f5d8 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:01.964370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b6904aef-173d-42f6-a2de-c75e4f31f5d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9774c354b8b048768f2071acfb48a4bf', 'request_id': 'req-b6904aef-173d-42f6-a2de-c75e4f31f5d8'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:02.085352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b6904aef-173d-42f6-a2de-c75e4f31f5d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9774c354b8b048768f2071acfb48a4bf', 'request_id': 'req-b6904aef-173d-42f6-a2de-c75e4f31f5d8'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:02.092691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b6904aef-173d-42f6-a2de-c75e4f31f5d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9774c354b8b048768f2071acfb48a4bf', 'request_id': 'req-b6904aef-173d-42f6-a2de-c75e4f31f5d8'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:02.107596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b6904aef-173d-42f6-a2de-c75e4f31f5d8 admin admin] The token cache is being invalidate because project 9774c354b8b048768f2071acfb48a4bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:31:02.108287 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b6904aef-173d-42f6-a2de-c75e4f31f5d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:02.109499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b6904aef-173d-42f6-a2de-c75e4f31f5d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:02.112271 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1172/2335] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:31:01 2022] DELETE /identity/v3/projects/9774c354b8b048768f2071acfb48a4bf => generated 0 bytes in 423 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:31:02.119187 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4fc4ab-ff93-4343-ab3f-d7be24b5d908 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:02.123151 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4fc4ab-ff93-4343-ab3f-d7be24b5d908 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:02.123703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4fc4ab-ff93-4343-ab3f-d7be24b5d908 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:02.147716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-2c4fc4ab-ff93-4343-ab3f-d7be24b5d908 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:31:02.150435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-2c4fc4ab-ff93-4343-ab3f-d7be24b5d908 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:02.220504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1164/2336] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:31:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:31:02.230705 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b517a353-1f22-48fd-a41e-97a97c64b02e None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:02.231538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b517a353-1f22-48fd-a41e-97a97c64b02e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:02.231853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b517a353-1f22-48fd-a41e-97a97c64b02e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:02.256373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-b517a353-1f22-48fd-a41e-97a97c64b02e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:31:02.257576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-b517a353-1f22-48fd-a41e-97a97c64b02e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:02.267387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1173/2337] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:31:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:31:02.272095 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4fc4ab-ff93-4343-ab3f-d7be24b5d908 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:02.281950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4fc4ab-ff93-4343-ab3f-d7be24b5d908 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:02.286860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d39762-9fee-4f6f-b678-7a9c1370d6a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:02.287464 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d39762-9fee-4f6f-b678-7a9c1370d6a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:02.287739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d39762-9fee-4f6f-b678-7a9c1370d6a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:02.288009 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d39762-9fee-4f6f-b678-7a9c1370d6a0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:02.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d39762-9fee-4f6f-b678-7a9c1370d6a0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:31:02.289147 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d39762-9fee-4f6f-b678-7a9c1370d6a0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:02.290773 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d39762-9fee-4f6f-b678-7a9c1370d6a0 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:02.304586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1165/2338] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:31:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:31:02.310105 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b517a353-1f22-48fd-a41e-97a97c64b02e None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:02.321409 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b517a353-1f22-48fd-a41e-97a97c64b02e None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:02.326704 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55159258-0f05-461f-ab8d-da5ab5dc9b7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:02.327353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55159258-0f05-461f-ab8d-da5ab5dc9b7b admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:02.327631 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55159258-0f05-461f-ab8d-da5ab5dc9b7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:02.327903 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55159258-0f05-461f-ab8d-da5ab5dc9b7b admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:02.328573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55159258-0f05-461f-ab8d-da5ab5dc9b7b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:02.330490 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55159258-0f05-461f-ab8d-da5ab5dc9b7b admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:02.474343 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1174/2339] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:31:02 2022] POST /identity/v3/projects => generated 383 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:31:02.478377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d39762-9fee-4f6f-b678-7a9c1370d6a0 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:02.489062 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d39762-9fee-4f6f-b678-7a9c1370d6a0 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:02.535366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:31:02.555840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0188150-1efb-4283-97f9-8f4d875966b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:02.555840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0188150-1efb-4283-97f9-8f4d875966b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:02.555840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0188150-1efb-4283-97f9-8f4d875966b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:02.555840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0188150-1efb-4283-97f9-8f4d875966b9 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:02.556279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0188150-1efb-4283-97f9-8f4d875966b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:02.556279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0188150-1efb-4283-97f9-8f4d875966b9 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:02.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1166/2340] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:31:02 2022] POST /identity/v3/users => generated 292 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:31:02.685076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55159258-0f05-461f-ab8d-da5ab5dc9b7b admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:02.696724 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55159258-0f05-461f-ab8d-da5ab5dc9b7b admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:02.703356 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999ddba9-fbc4-4694-8a05-36927d95f6cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:02.707448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999ddba9-fbc4-4694-8a05-36927d95f6cf admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:02.707448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999ddba9-fbc4-4694-8a05-36927d95f6cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:02.707448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999ddba9-fbc4-4694-8a05-36927d95f6cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:02.707448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999ddba9-fbc4-4694-8a05-36927d95f6cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:02.707448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999ddba9-fbc4-4694-8a05-36927d95f6cf admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:02.722674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1175/2341] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:31:02 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:31:02.733196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0188150-1efb-4283-97f9-8f4d875966b9 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:02.738558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0188150-1efb-4283-97f9-8f4d875966b9 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:02.745913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cc2c76-5994-4518-a942-8eafea50b7e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:02.746587 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cc2c76-5994-4518-a942-8eafea50b7e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:02.746867 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cc2c76-5994-4518-a942-8eafea50b7e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:02.747135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cc2c76-5994-4518-a942-8eafea50b7e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:02.747774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cc2c76-5994-4518-a942-8eafea50b7e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:02.749433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cc2c76-5994-4518-a942-8eafea50b7e8 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:02.760441 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1167/2342] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:31:02 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:31:02.764679 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999ddba9-fbc4-4694-8a05-36927d95f6cf admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:02.774876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999ddba9-fbc4-4694-8a05-36927d95f6cf admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:02.779928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8650b711-00fb-479e-89a3-3aa915e2e12f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:02.780618 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8650b711-00fb-479e-89a3-3aa915e2e12f admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:02.780895 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8650b711-00fb-479e-89a3-3aa915e2e12f admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:02.781163 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8650b711-00fb-479e-89a3-3aa915e2e12f admin admin] PATH_INFO: `/v3/projects/122b46d9e29843a88c7ca5f019426539/users/a91ac0ba7e8c4418930852b0fa184247/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:02.796917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8650b711-00fb-479e-89a3-3aa915e2e12f admin admin] RBAC: Authorizing `identity:create_grant(project_id=122b46d9e29843a88c7ca5f019426539, user_id=a91ac0ba7e8c4418930852b0fa184247, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:02.799573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8650b711-00fb-479e-89a3-3aa915e2e12f admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:02.887435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1176/2343] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:31:02 2022] PUT /identity/v3/projects/122b46d9e29843a88c7ca5f019426539/users/a91ac0ba7e8c4418930852b0fa184247/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:31:02.896995 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd801a80-adf4-4ce4-9011-0cba281d1973 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:02.897279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd801a80-adf4-4ce4-9011-0cba281d1973 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:02.897550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd801a80-adf4-4ce4-9011-0cba281d1973 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:02.916941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-cd801a80-adf4-4ce4-9011-0cba281d1973 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:31:02.920734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-cd801a80-adf4-4ce4-9011-0cba281d1973 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:02.990036 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1168/2344] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:31:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:31:03.000509 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8650b711-00fb-479e-89a3-3aa915e2e12f admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:03.002230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-8650b711-00fb-479e-89a3-3aa915e2e12f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:03.021938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:31:03.065404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8650b711-00fb-479e-89a3-3aa915e2e12f admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:03.069326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449090fd-d21f-4e89-ae29-4d33e26e9b93 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:03.069864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449090fd-d21f-4e89-ae29-4d33e26e9b93 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:03.070141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449090fd-d21f-4e89-ae29-4d33e26e9b93 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:03.070412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449090fd-d21f-4e89-ae29-4d33e26e9b93 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:03.081873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449090fd-d21f-4e89-ae29-4d33e26e9b93 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:03.083344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449090fd-d21f-4e89-ae29-4d33e26e9b93 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:03.095643 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1177/2345] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:31:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:31:09.007959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502c2487-270b-41b2-9475-82f1c2356563 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:09.010321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502c2487-270b-41b2-9475-82f1c2356563 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:09.010321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502c2487-270b-41b2-9475-82f1c2356563 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:09.040486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-502c2487-270b-41b2-9475-82f1c2356563 None None] MFA Rules not processed for user `a91ac0ba7e8c4418930852b0fa184247`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:31:09.041715 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-502c2487-270b-41b2-9475-82f1c2356563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:09.118772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1169/2346] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:31:09 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:31:09.134662 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449090fd-d21f-4e89-ae29-4d33e26e9b93 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:09.136925 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-449090fd-d21f-4e89-ae29-4d33e26e9b93 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:09.221859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449090fd-d21f-4e89-ae29-4d33e26e9b93 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:09.227796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9bcc522-6573-4a92-9f55-0d4338fe1aff service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:09.228462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bcc522-6573-4a92-9f55-0d4338fe1aff service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:09.228740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bcc522-6573-4a92-9f55-0d4338fe1aff service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:09.229011 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bcc522-6573-4a92-9f55-0d4338fe1aff service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:09.238366 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9bcc522-6573-4a92-9f55-0d4338fe1aff service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:09.239973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9bcc522-6573-4a92-9f55-0d4338fe1aff service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:09.249752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1178/2347] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:31:09 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:31:11.800259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343512ba-774b-4bd3-9dc9-f78c4fecd01c None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:11.800898 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343512ba-774b-4bd3-9dc9-f78c4fecd01c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:11.801277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343512ba-774b-4bd3-9dc9-f78c4fecd01c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:11.822038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-343512ba-774b-4bd3-9dc9-f78c4fecd01c None None] MFA Rules not processed for user `a91ac0ba7e8c4418930852b0fa184247`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:31:11.823261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-343512ba-774b-4bd3-9dc9-f78c4fecd01c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:11.832141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1170/2348] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:31:11 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:31:13.646304 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2a235c-d238-45dc-80dd-9bf26a296b5f None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:13.647058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2a235c-d238-45dc-80dd-9bf26a296b5f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:13.647234 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2a235c-d238-45dc-80dd-9bf26a296b5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:13.667122 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-bc2a235c-d238-45dc-80dd-9bf26a296b5f None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:31:13.668975 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-bc2a235c-d238-45dc-80dd-9bf26a296b5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:13.731179 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1179/2349] 10.222.0.47 () {58 vars in 967 bytes} [Sun Aug 21 01:31:13 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:31:13.911825 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343512ba-774b-4bd3-9dc9-f78c4fecd01c None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:13.913307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-343512ba-774b-4bd3-9dc9-f78c4fecd01c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:13.991907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:31:13.999271 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343512ba-774b-4bd3-9dc9-f78c4fecd01c None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:14.008355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594e5b05-b457-48ae-ac63-9ca1ff34c80e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:14.009352 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e5b05-b457-48ae-ac63-9ca1ff34c80e service swift] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:14.009692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e5b05-b457-48ae-ac63-9ca1ff34c80e service swift] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:14.010086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e5b05-b457-48ae-ac63-9ca1ff34c80e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:14.025481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594e5b05-b457-48ae-ac63-9ca1ff34c80e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:14.028059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594e5b05-b457-48ae-ac63-9ca1ff34c80e service swift] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:14.037481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1171/2350] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:31:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 21 01:31:23.418920 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2a235c-d238-45dc-80dd-9bf26a296b5f None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:23.420910 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-bc2a235c-d238-45dc-80dd-9bf26a296b5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:23.437726 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2a235c-d238-45dc-80dd-9bf26a296b5f None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:23.443925 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5827720a-07ef-4716-9a95-47ece7a5ecd0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:23.444569 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5827720a-07ef-4716-9a95-47ece7a5ecd0 service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:23.444847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5827720a-07ef-4716-9a95-47ece7a5ecd0 service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:23.445116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5827720a-07ef-4716-9a95-47ece7a5ecd0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:23.446917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-5827720a-07ef-4716-9a95-47ece7a5ecd0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:23.461243 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5827720a-07ef-4716-9a95-47ece7a5ecd0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:23.462866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5827720a-07ef-4716-9a95-47ece7a5ecd0 service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:23.475045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1180/2351] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:31:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:31:24.282705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=10 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:31:24.766313 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594e5b05-b457-48ae-ac63-9ca1ff34c80e service swift] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:24.767837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-594e5b05-b457-48ae-ac63-9ca1ff34c80e service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:24.782443 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594e5b05-b457-48ae-ac63-9ca1ff34c80e service swift] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:24.787096 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c2c2f4-03a5-45ae-9fdd-bd331dd80efb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:24.787794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c2c2f4-03a5-45ae-9fdd-bd331dd80efb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:24.791464 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c2c2f4-03a5-45ae-9fdd-bd331dd80efb admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:24.791464 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c2c2f4-03a5-45ae-9fdd-bd331dd80efb admin admin] PATH_INFO: `/v3/users/2325a3d547d043d2a9ac95986187027b` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:24.791464 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c2c2f4-03a5-45ae-9fdd-bd331dd80efb admin admin] RBAC: Authorizing `identity:delete_user(user_id=2325a3d547d043d2a9ac95986187027b)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:24.791464 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c2c2f4-03a5-45ae-9fdd-bd331dd80efb admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:24.864261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5827720a-07ef-4716-9a95-47ece7a5ecd0 service nova] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:24.865767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-5827720a-07ef-4716-9a95-47ece7a5ecd0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:24.880208 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5827720a-07ef-4716-9a95-47ece7a5ecd0 service nova] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:24.885415 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b9f415-80b1-43f4-9b81-c0ddc6b3f2ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:24.894123 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b9f415-80b1-43f4-9b81-c0ddc6b3f2ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:24.894123 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b9f415-80b1-43f4-9b81-c0ddc6b3f2ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:24.894123 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b9f415-80b1-43f4-9b81-c0ddc6b3f2ae admin admin] PATH_INFO: `/v3/users/2a6687a3b09445d897879095287a8586` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:24.894123 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b9f415-80b1-43f4-9b81-c0ddc6b3f2ae admin admin] RBAC: Authorizing `identity:delete_user(user_id=2a6687a3b09445d897879095287a8586)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:24.894123 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b9f415-80b1-43f4-9b81-c0ddc6b3f2ae admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:25.127888 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:31:25.140327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c8c2c2f4-03a5-45ae-9fdd-bd331dd80efb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2325a3d547d043d2a9ac95986187027b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:25.147250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c8c2c2f4-03a5-45ae-9fdd-bd331dd80efb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2325a3d547d043d2a9ac95986187027b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:25.147927 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c8c2c2f4-03a5-45ae-9fdd-bd331dd80efb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2325a3d547d043d2a9ac95986187027b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:25.160020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c8c2c2f4-03a5-45ae-9fdd-bd331dd80efb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2325a3d547d043d2a9ac95986187027b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:25.215983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e4b9f415-80b1-43f4-9b81-c0ddc6b3f2ae admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2a6687a3b09445d897879095287a8586'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:25.223404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e4b9f415-80b1-43f4-9b81-c0ddc6b3f2ae admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2a6687a3b09445d897879095287a8586'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:25.235426 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e4b9f415-80b1-43f4-9b81-c0ddc6b3f2ae admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2a6687a3b09445d897879095287a8586'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:25.267024 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c8c2c2f4-03a5-45ae-9fdd-bd331dd80efb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2325a3d547d043d2a9ac95986187027b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:25.273393 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1172/2352] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:31:24 2022] DELETE /identity/v3/users/2325a3d547d043d2a9ac95986187027b => generated 0 bytes in 508 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:31:25.318617 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e4b9f415-80b1-43f4-9b81-c0ddc6b3f2ae admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2a6687a3b09445d897879095287a8586'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:25.319654 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e4b9f415-80b1-43f4-9b81-c0ddc6b3f2ae admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2a6687a3b09445d897879095287a8586'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:25.326983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1181/2353] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:31:24 2022] DELETE /identity/v3/users/2a6687a3b09445d897879095287a8586 => generated 0 bytes in 463 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:31:25.571666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c2c2f4-03a5-45ae-9fdd-bd331dd80efb admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:25.573844 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-c8c2c2f4-03a5-45ae-9fdd-bd331dd80efb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:25.683922 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c2c2f4-03a5-45ae-9fdd-bd331dd80efb admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:25.738302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b9f415-80b1-43f4-9b81-c0ddc6b3f2ae admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:25.740010 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e4b9f415-80b1-43f4-9b81-c0ddc6b3f2ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:25.748635 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23211f91-d5df-498d-a7c2-cff9d96c8a6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:25.749311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23211f91-d5df-498d-a7c2-cff9d96c8a6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:25.749472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23211f91-d5df-498d-a7c2-cff9d96c8a6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:25.749743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23211f91-d5df-498d-a7c2-cff9d96c8a6b admin admin] PATH_INFO: `/v3/projects/67f492f96d314cfea70bdbfa3caaee9e` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:25.750998 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23211f91-d5df-498d-a7c2-cff9d96c8a6b admin admin] RBAC: Authorizing `identity:delete_project(project_id=67f492f96d314cfea70bdbfa3caaee9e)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:25.752984 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23211f91-d5df-498d-a7c2-cff9d96c8a6b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:25.790911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:31:25.822208 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b9f415-80b1-43f4-9b81-c0ddc6b3f2ae admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:25.827355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3668ad0-ae93-4fbd-97dc-f1287cea7524 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:25.827982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3668ad0-ae93-4fbd-97dc-f1287cea7524 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:25.828310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3668ad0-ae93-4fbd-97dc-f1287cea7524 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:25.828583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3668ad0-ae93-4fbd-97dc-f1287cea7524 admin admin] PATH_INFO: `/v3/projects/da8fc458ef6b49e18daac9fb195a2149` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:25.829700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3668ad0-ae93-4fbd-97dc-f1287cea7524 admin admin] RBAC: Authorizing `identity:delete_project(project_id=da8fc458ef6b49e18daac9fb195a2149)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:25.831717 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3668ad0-ae93-4fbd-97dc-f1287cea7524 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:25.880333 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-23211f91-d5df-498d-a7c2-cff9d96c8a6b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '67f492f96d314cfea70bdbfa3caaee9e', 'request_id': 'req-23211f91-d5df-498d-a7c2-cff9d96c8a6b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:25.886101 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-23211f91-d5df-498d-a7c2-cff9d96c8a6b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '67f492f96d314cfea70bdbfa3caaee9e', 'request_id': 'req-23211f91-d5df-498d-a7c2-cff9d96c8a6b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:25.892559 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-23211f91-d5df-498d-a7c2-cff9d96c8a6b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '67f492f96d314cfea70bdbfa3caaee9e', 'request_id': 'req-23211f91-d5df-498d-a7c2-cff9d96c8a6b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:26.029123 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-a3668ad0-ae93-4fbd-97dc-f1287cea7524 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da8fc458ef6b49e18daac9fb195a2149', 'request_id': 'req-a3668ad0-ae93-4fbd-97dc-f1287cea7524'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:26.046027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-23211f91-d5df-498d-a7c2-cff9d96c8a6b admin admin] The token cache is being invalidate because project 67f492f96d314cfea70bdbfa3caaee9e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:31:26.046027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-23211f91-d5df-498d-a7c2-cff9d96c8a6b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:26.046027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-23211f91-d5df-498d-a7c2-cff9d96c8a6b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:26.046027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1173/2354] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:31:25 2022] DELETE /identity/v3/projects/67f492f96d314cfea70bdbfa3caaee9e => generated 0 bytes in 472 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:31:26.124248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-a3668ad0-ae93-4fbd-97dc-f1287cea7524 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da8fc458ef6b49e18daac9fb195a2149', 'request_id': 'req-a3668ad0-ae93-4fbd-97dc-f1287cea7524'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:26.131227 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-a3668ad0-ae93-4fbd-97dc-f1287cea7524 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da8fc458ef6b49e18daac9fb195a2149', 'request_id': 'req-a3668ad0-ae93-4fbd-97dc-f1287cea7524'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:26.144042 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-a3668ad0-ae93-4fbd-97dc-f1287cea7524 admin admin] The token cache is being invalidate because project da8fc458ef6b49e18daac9fb195a2149 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:31:26.144397 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-a3668ad0-ae93-4fbd-97dc-f1287cea7524 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:26.145101 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-a3668ad0-ae93-4fbd-97dc-f1287cea7524 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:26.146723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1182/2355] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:31:25 2022] DELETE /identity/v3/projects/da8fc458ef6b49e18daac9fb195a2149 => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:31:26.151088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23211f91-d5df-498d-a7c2-cff9d96c8a6b admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:26.154504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-23211f91-d5df-498d-a7c2-cff9d96c8a6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:26.227860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23211f91-d5df-498d-a7c2-cff9d96c8a6b admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:26.285695 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcc1227-c907-4f12-bc98-09668d764563 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:26.286246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcc1227-c907-4f12-bc98-09668d764563 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:26.286521 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcc1227-c907-4f12-bc98-09668d764563 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:26.286791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcc1227-c907-4f12-bc98-09668d764563 admin admin] PATH_INFO: `/v3/users/d6fca69450794a2bb832e658fc1f6861` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:26.287911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcc1227-c907-4f12-bc98-09668d764563 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d6fca69450794a2bb832e658fc1f6861)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:26.289873 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcc1227-c907-4f12-bc98-09668d764563 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:26.495450 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6bcc1227-c907-4f12-bc98-09668d764563 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd6fca69450794a2bb832e658fc1f6861'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:26.502447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6bcc1227-c907-4f12-bc98-09668d764563 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd6fca69450794a2bb832e658fc1f6861'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:26.503132 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6bcc1227-c907-4f12-bc98-09668d764563 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd6fca69450794a2bb832e658fc1f6861'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:26.509894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:31:26.517625 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6bcc1227-c907-4f12-bc98-09668d764563 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd6fca69450794a2bb832e658fc1f6861'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:26.601023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6bcc1227-c907-4f12-bc98-09668d764563 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd6fca69450794a2bb832e658fc1f6861'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:26.607565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1174/2356] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:31:26 2022] DELETE /identity/v3/users/d6fca69450794a2bb832e658fc1f6861 => generated 0 bytes in 457 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:31:26.899979 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3668ad0-ae93-4fbd-97dc-f1287cea7524 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:26.901850 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-a3668ad0-ae93-4fbd-97dc-f1287cea7524 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:26.931008 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3668ad0-ae93-4fbd-97dc-f1287cea7524 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:26.969878 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:31:26.995494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:26.996153 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:26.996431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:26.996697 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd admin admin] PATH_INFO: `/v3/projects/a7696238a2fa45afb6005154de8a14e8` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:26.997845 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7696238a2fa45afb6005154de8a14e8)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:27.000705 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:27.150696 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7696238a2fa45afb6005154de8a14e8', 'request_id': 'req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:27.252405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7696238a2fa45afb6005154de8a14e8', 'request_id': 'req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:27.259535 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7696238a2fa45afb6005154de8a14e8', 'request_id': 'req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:27.271326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd admin admin] The token cache is being invalidate because project a7696238a2fa45afb6005154de8a14e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:31:27.271613 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:27.272305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:31:27.273858 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1183/2357] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:31:26 2022] DELETE /identity/v3/projects/a7696238a2fa45afb6005154de8a14e8 => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:31:32.174995 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fa39ff-0679-40e5-9d94-05c800ed2b7c None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:32.175488 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fa39ff-0679-40e5-9d94-05c800ed2b7c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:32.175586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fa39ff-0679-40e5-9d94-05c800ed2b7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:32.196209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-e4fa39ff-0679-40e5-9d94-05c800ed2b7c None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:31:32.197628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e4fa39ff-0679-40e5-9d94-05c800ed2b7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:32.258650 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1175/2358] 10.222.0.47 () {58 vars in 1009 bytes} [Sun Aug 21 01:31:32 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:31:32.270670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:32.272144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:32.342586 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a44f41-5c3c-4b3e-b5e7-3cfbfa228dbd admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:32.348467 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0774763c-f4c6-419d-916d-2d8c1d163b57 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:32.349079 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0774763c-f4c6-419d-916d-2d8c1d163b57 service swift] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:32.359376 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0774763c-f4c6-419d-916d-2d8c1d163b57 service swift] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:32.359376 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0774763c-f4c6-419d-916d-2d8c1d163b57 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:32.359376 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0774763c-f4c6-419d-916d-2d8c1d163b57 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:32.360655 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0774763c-f4c6-419d-916d-2d8c1d163b57 service swift] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:32.369250 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1184/2359] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:31:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 21 01:31:42.251899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=32,keystone:DELETE=3,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:31:42.365680 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=67,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:31:50.420285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fa39ff-0679-40e5-9d94-05c800ed2b7c None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:50.421851 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e4fa39ff-0679-40e5-9d94-05c800ed2b7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:50.553640 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fa39ff-0679-40e5-9d94-05c800ed2b7c None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:50.559581 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fcf690-a476-426c-809e-f2c00fa00afb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:50.560622 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fcf690-a476-426c-809e-f2c00fa00afb service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:50.560911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fcf690-a476-426c-809e-f2c00fa00afb service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:50.561433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fcf690-a476-426c-809e-f2c00fa00afb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:50.564067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-76fcf690-a476-426c-809e-f2c00fa00afb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:50.583690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fcf690-a476-426c-809e-f2c00fa00afb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:50.583690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fcf690-a476-426c-809e-f2c00fa00afb service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:50.589242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1176/2360] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:31:50 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:31:51.154220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedba59a-e3bf-49e4-abc6-c91d45e27989 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:51.154976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedba59a-e3bf-49e4-abc6-c91d45e27989 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:51.154976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedba59a-e3bf-49e4-abc6-c91d45e27989 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:51.156203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1185/2361] 10.222.0.47 () {54 vars in 921 bytes} [Sun Aug 21 01:31:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:31:51.164145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efed00a5-2316-48ea-9cdb-bc961baa9630 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:51.164465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efed00a5-2316-48ea-9cdb-bc961baa9630 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:51.164687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efed00a5-2316-48ea-9cdb-bc961baa9630 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:51.252913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-efed00a5-2316-48ea-9cdb-bc961baa9630 None None] MFA Rules not processed for user `fabdcdc25d1d4ef6bc66bd6cadecaaf8`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:31:51.255099 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-efed00a5-2316-48ea-9cdb-bc961baa9630 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:51.389829 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1177/2362] 10.222.0.47 () {58 vars in 1018 bytes} [Sun Aug 21 01:31:51 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:31:51.402808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedba59a-e3bf-49e4-abc6-c91d45e27989 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:51.404622 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-fedba59a-e3bf-49e4-abc6-c91d45e27989 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:51.463826 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedba59a-e3bf-49e4-abc6-c91d45e27989 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:51.467421 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64e6079-5dfa-4d4e-b10a-4f5245d0ef39 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6365b2328f34ab9b1b73ececbf5cc30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:51.468013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64e6079-5dfa-4d4e-b10a-4f5245d0ef39 service glance] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:51.468316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64e6079-5dfa-4d4e-b10a-4f5245d0ef39 service glance] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:51.468586 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64e6079-5dfa-4d4e-b10a-4f5245d0ef39 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:51.479880 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64e6079-5dfa-4d4e-b10a-4f5245d0ef39 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:51.481332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64e6079-5dfa-4d4e-b10a-4f5245d0ef39 service glance] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:51.492015 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1186/2363] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:31:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:31:51.786774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d07cc4-281f-4732-ad63-4ee7d0639e3e None None] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:51.787360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d07cc4-281f-4732-ad63-4ee7d0639e3e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:51.787360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d07cc4-281f-4732-ad63-4ee7d0639e3e None None] PATH_INFO: `/` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:51.788680 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1178/2364] 10.222.0.47 () {54 vars in 921 bytes} [Sun Aug 21 01:31:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:31:51.793607 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd09902d-056c-461c-ae08-d1a23c1d61c4 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:51.793885 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd09902d-056c-461c-ae08-d1a23c1d61c4 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:51.794153 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd09902d-056c-461c-ae08-d1a23c1d61c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:51.816011 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-bd09902d-056c-461c-ae08-d1a23c1d61c4 None None] MFA Rules not processed for user `b532ddab1f5e467989edaa9e4b74bd72`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:31:51.823868 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-bd09902d-056c-461c-ae08-d1a23c1d61c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:51.824585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1187/2365] 10.222.0.47 () {58 vars in 1018 bytes} [Sun Aug 21 01:31:51 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:31:51.838620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d07cc4-281f-4732-ad63-4ee7d0639e3e None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:51.853975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d07cc4-281f-4732-ad63-4ee7d0639e3e None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:51.857673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b685bf6b-5498-4fae-99df-8e8573c9524d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:51.858269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b685bf6b-5498-4fae-99df-8e8573c9524d service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:51.858542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b685bf6b-5498-4fae-99df-8e8573c9524d service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:51.858808 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b685bf6b-5498-4fae-99df-8e8573c9524d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:51.864617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:31:51.869520 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b685bf6b-5498-4fae-99df-8e8573c9524d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:51.869520 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b685bf6b-5498-4fae-99df-8e8573c9524d service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:51.878053 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1179/2366] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:31:51 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:31:51.977800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c435dca-13b9-461f-81bf-7f18eb356a80 None None] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:51.978082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c435dca-13b9-461f-81bf-7f18eb356a80 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:51.978360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c435dca-13b9-461f-81bf-7f18eb356a80 None None] PATH_INFO: `/` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:51.979724 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1188/2367] 10.222.0.47 () {54 vars in 921 bytes} [Sun Aug 21 01:31:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 21 01:31:51.984920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c6d495-3a07-414d-b78e-ad1861da247b None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:51.985211 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c6d495-3a07-414d-b78e-ad1861da247b None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:51.985487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c6d495-3a07-414d-b78e-ad1861da247b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:52.011131 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-47c6d495-3a07-414d-b78e-ad1861da247b None None] MFA Rules not processed for user `fabdcdc25d1d4ef6bc66bd6cadecaaf8`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:31:52.012609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-47c6d495-3a07-414d-b78e-ad1861da247b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:52.023047 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1180/2368] 10.222.0.47 () {58 vars in 1018 bytes} [Sun Aug 21 01:31:51 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:31:52.042467 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c435dca-13b9-461f-81bf-7f18eb356a80 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:52.055502 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c435dca-13b9-461f-81bf-7f18eb356a80 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:52.059138 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b981cef5-4312-4a93-b4ea-8b15948dc4af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:52.063798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b981cef5-4312-4a93-b4ea-8b15948dc4af service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:52.063798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b981cef5-4312-4a93-b4ea-8b15948dc4af service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:52.063798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b981cef5-4312-4a93-b4ea-8b15948dc4af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:52.079529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b981cef5-4312-4a93-b4ea-8b15948dc4af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:52.081573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b981cef5-4312-4a93-b4ea-8b15948dc4af service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:52.094159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1189/2369] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:31:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:31:57.501200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6d4b44-5796-49c6-8f55-c50e4de860aa None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:57.502049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6d4b44-5796-49c6-8f55-c50e4de860aa None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:57.502049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6d4b44-5796-49c6-8f55-c50e4de860aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:57.530487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-cc6d4b44-5796-49c6-8f55-c50e4de860aa None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:31:57.531975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-cc6d4b44-5796-49c6-8f55-c50e4de860aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:31:57.540608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1181/2370] 10.222.0.47 () {58 vars in 1009 bytes} [Sun Aug 21 01:31:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:31:57.553758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b981cef5-4312-4a93-b4ea-8b15948dc4af service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:31:57.573688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b981cef5-4312-4a93-b4ea-8b15948dc4af service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:31:57.586531 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4484f5b7-86f1-46af-9b1f-9d2114b86dcd service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:31:57.587417 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4484f5b7-86f1-46af-9b1f-9d2114b86dcd service swift] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:31:57.587883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4484f5b7-86f1-46af-9b1f-9d2114b86dcd service swift] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:31:57.588334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4484f5b7-86f1-46af-9b1f-9d2114b86dcd service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:31:57.601148 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4484f5b7-86f1-46af-9b1f-9d2114b86dcd service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:31:57.603111 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4484f5b7-86f1-46af-9b1f-9d2114b86dcd service swift] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:31:57.615941 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1190/2371] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:31:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 21 01:32:07.525207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=8 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:07.611078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=41 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:16.047281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba61a1a-ebd2-456f-9de2-bdba50f5ed1d None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:16.047814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba61a1a-ebd2-456f-9de2-bdba50f5ed1d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:16.047913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba61a1a-ebd2-456f-9de2-bdba50f5ed1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:16.068369 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-0ba61a1a-ebd2-456f-9de2-bdba50f5ed1d None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:16.069576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-0ba61a1a-ebd2-456f-9de2-bdba50f5ed1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:16.076886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1182/2372] 10.222.0.47 () {58 vars in 967 bytes} [Sun Aug 21 01:32:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:16.099993 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4484f5b7-86f1-46af-9b1f-9d2114b86dcd service swift] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:16.110589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4484f5b7-86f1-46af-9b1f-9d2114b86dcd service swift] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:16.115686 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40525996-0f6b-473f-bb75-bc2779bafcc9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:16.116309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40525996-0f6b-473f-bb75-bc2779bafcc9 service swift] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:16.116585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40525996-0f6b-473f-bb75-bc2779bafcc9 service swift] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:16.116851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40525996-0f6b-473f-bb75-bc2779bafcc9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:16.125191 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40525996-0f6b-473f-bb75-bc2779bafcc9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:16.126751 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40525996-0f6b-473f-bb75-bc2779bafcc9 service swift] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:16.134901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1191/2373] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:32:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 21 01:32:25.826030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba61a1a-ebd2-456f-9de2-bdba50f5ed1d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:25.827503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-0ba61a1a-ebd2-456f-9de2-bdba50f5ed1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:25.839838 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba61a1a-ebd2-456f-9de2-bdba50f5ed1d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:25.843523 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb7a0bd-6699-4f04-ab35-f06c229dd344 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:25.844170 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb7a0bd-6699-4f04-ab35-f06c229dd344 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:25.844454 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb7a0bd-6699-4f04-ab35-f06c229dd344 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:25.844723 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb7a0bd-6699-4f04-ab35-f06c229dd344 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:25.846453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-4fb7a0bd-6699-4f04-ab35-f06c229dd344 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:25.936835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb7a0bd-6699-4f04-ab35-f06c229dd344 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:25.938249 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb7a0bd-6699-4f04-ab35-f06c229dd344 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:25.958256 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1183/2374] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:32:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:32:26.131499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=6 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:29.522519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40525996-0f6b-473f-bb75-bc2779bafcc9 service swift] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:29.524567 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-40525996-0f6b-473f-bb75-bc2779bafcc9 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:29.597569 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40525996-0f6b-473f-bb75-bc2779bafcc9 service swift] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:29.603431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a262154c-7c95-4791-94f4-4b2563415c35 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '972b0f1dbfef4d85a6dfc4c5fd839bde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:29.604038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a262154c-7c95-4791-94f4-4b2563415c35 service placement] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:29.604344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a262154c-7c95-4791-94f4-4b2563415c35 service placement] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:29.604615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a262154c-7c95-4791-94f4-4b2563415c35 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:29.606380 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-a262154c-7c95-4791-94f4-4b2563415c35 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:29.617865 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a262154c-7c95-4791-94f4-4b2563415c35 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:29.619440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a262154c-7c95-4791-94f4-4b2563415c35 service placement] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:29.629334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1192/2375] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:32:29 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:32:29.660351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb7a0bd-6699-4f04-ab35-f06c229dd344 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:29.661812 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-4fb7a0bd-6699-4f04-ab35-f06c229dd344 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:29.675389 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb7a0bd-6699-4f04-ab35-f06c229dd344 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:29.680278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba2f50e-04c5-40cf-910d-81121f0e93e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:29.680911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba2f50e-04c5-40cf-910d-81121f0e93e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:29.681346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba2f50e-04c5-40cf-910d-81121f0e93e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:29.681628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba2f50e-04c5-40cf-910d-81121f0e93e3 admin admin] PATH_INFO: `/v3/users/0cd8d822f15943a78575c3744692cdb9` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:29.682791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba2f50e-04c5-40cf-910d-81121f0e93e3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0cd8d822f15943a78575c3744692cdb9)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:29.684742 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba2f50e-04c5-40cf-910d-81121f0e93e3 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:29.867547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5ba2f50e-04c5-40cf-910d-81121f0e93e3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0cd8d822f15943a78575c3744692cdb9'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:29.874648 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5ba2f50e-04c5-40cf-910d-81121f0e93e3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0cd8d822f15943a78575c3744692cdb9'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:29.875326 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5ba2f50e-04c5-40cf-910d-81121f0e93e3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0cd8d822f15943a78575c3744692cdb9'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:29.888439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5ba2f50e-04c5-40cf-910d-81121f0e93e3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0cd8d822f15943a78575c3744692cdb9'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:30.002413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5ba2f50e-04c5-40cf-910d-81121f0e93e3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0cd8d822f15943a78575c3744692cdb9'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:30.009456 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1184/2376] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:32:29 2022] DELETE /identity/v3/users/0cd8d822f15943a78575c3744692cdb9 => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:30.306472 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a262154c-7c95-4791-94f4-4b2563415c35 service placement] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:30.318152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a262154c-7c95-4791-94f4-4b2563415c35 service placement] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:30.396805 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3d5757-b2bd-4c53-bda8-21108eb45052 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:30.397499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3d5757-b2bd-4c53-bda8-21108eb45052 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:30.397773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3d5757-b2bd-4c53-bda8-21108eb45052 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:30.398125 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3d5757-b2bd-4c53-bda8-21108eb45052 admin admin] PATH_INFO: `/v3/projects/f4a0fcea08b248fe84012875a7243761` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:30.399494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3d5757-b2bd-4c53-bda8-21108eb45052 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f4a0fcea08b248fe84012875a7243761)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:30.401610 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3d5757-b2bd-4c53-bda8-21108eb45052 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:30.518193 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7b3d5757-b2bd-4c53-bda8-21108eb45052 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f4a0fcea08b248fe84012875a7243761', 'request_id': 'req-7b3d5757-b2bd-4c53-bda8-21108eb45052'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:30.584963 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7b3d5757-b2bd-4c53-bda8-21108eb45052 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f4a0fcea08b248fe84012875a7243761', 'request_id': 'req-7b3d5757-b2bd-4c53-bda8-21108eb45052'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:30.590982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7b3d5757-b2bd-4c53-bda8-21108eb45052 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f4a0fcea08b248fe84012875a7243761', 'request_id': 'req-7b3d5757-b2bd-4c53-bda8-21108eb45052'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:30.602745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7b3d5757-b2bd-4c53-bda8-21108eb45052 admin admin] The token cache is being invalidate because project f4a0fcea08b248fe84012875a7243761 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:32:30.603060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7b3d5757-b2bd-4c53-bda8-21108eb45052 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:30.603692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7b3d5757-b2bd-4c53-bda8-21108eb45052 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:30.611282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1193/2377] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:32:30 2022] DELETE /identity/v3/projects/f4a0fcea08b248fe84012875a7243761 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:32:30.611410 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2aefaa-4e47-4cdb-ab6f-80b78e49cb85 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:30.611410 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2aefaa-4e47-4cdb-ab6f-80b78e49cb85 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:30.611410 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2aefaa-4e47-4cdb-ab6f-80b78e49cb85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:30.658543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-eb2aefaa-4e47-4cdb-ab6f-80b78e49cb85 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:30.659729 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-eb2aefaa-4e47-4cdb-ab6f-80b78e49cb85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:30.663899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:30.721956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1185/2378] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:30.730559 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b0213e-916d-4f43-9ff3-81e50abd691b None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:30.730847 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b0213e-916d-4f43-9ff3-81e50abd691b None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:30.731152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b0213e-916d-4f43-9ff3-81e50abd691b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:30.752604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-e8b0213e-916d-4f43-9ff3-81e50abd691b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:30.753858 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e8b0213e-916d-4f43-9ff3-81e50abd691b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:30.761960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1194/2379] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:30.766294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2aefaa-4e47-4cdb-ab6f-80b78e49cb85 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:30.776085 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2aefaa-4e47-4cdb-ab6f-80b78e49cb85 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:30.780888 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51818ce-46ef-4103-8540-55a4e1af5dd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:30.781487 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51818ce-46ef-4103-8540-55a4e1af5dd1 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:30.781767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51818ce-46ef-4103-8540-55a4e1af5dd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:30.782032 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51818ce-46ef-4103-8540-55a4e1af5dd1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:30.782665 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51818ce-46ef-4103-8540-55a4e1af5dd1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:32:30.782958 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51818ce-46ef-4103-8540-55a4e1af5dd1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:30.784635 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51818ce-46ef-4103-8540-55a4e1af5dd1 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:30.797934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1186/2380] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:32:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:32:30.802365 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b0213e-916d-4f43-9ff3-81e50abd691b None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:30.815004 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b0213e-916d-4f43-9ff3-81e50abd691b None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:30.821742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fb8533-9d11-4299-9941-ac79356cc0d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:30.822445 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fb8533-9d11-4299-9941-ac79356cc0d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:30.822726 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fb8533-9d11-4299-9941-ac79356cc0d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:30.823001 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fb8533-9d11-4299-9941-ac79356cc0d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:30.823668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fb8533-9d11-4299-9941-ac79356cc0d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:30.825599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fb8533-9d11-4299-9941-ac79356cc0d2 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:30.841528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:30.928382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1195/2381] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:32:30 2022] POST /identity/v3/projects => generated 371 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:32:30.932597 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51818ce-46ef-4103-8540-55a4e1af5dd1 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:30.942774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51818ce-46ef-4103-8540-55a4e1af5dd1 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:30.999607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f13482-4e77-450d-91eb-82fb87896d5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:31.000244 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f13482-4e77-450d-91eb-82fb87896d5c admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:31.000523 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f13482-4e77-450d-91eb-82fb87896d5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:31.000791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f13482-4e77-450d-91eb-82fb87896d5c admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:31.001420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f13482-4e77-450d-91eb-82fb87896d5c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:31.003314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f13482-4e77-450d-91eb-82fb87896d5c admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:31.104750 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1187/2382] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:32:30 2022] POST /identity/v3/users => generated 286 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:32:31.109065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fb8533-9d11-4299-9941-ac79356cc0d2 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:31.119881 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fb8533-9d11-4299-9941-ac79356cc0d2 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:31.124986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd8d6c1-ebdc-4416-a218-75133411a444 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:31.125638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd8d6c1-ebdc-4416-a218-75133411a444 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:31.125910 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd8d6c1-ebdc-4416-a218-75133411a444 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:31.127221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd8d6c1-ebdc-4416-a218-75133411a444 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:31.127221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd8d6c1-ebdc-4416-a218-75133411a444 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:31.128634 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd8d6c1-ebdc-4416-a218-75133411a444 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:31.140450 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1196/2383] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:32:31 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:32:31.144480 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f13482-4e77-450d-91eb-82fb87896d5c admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:31.156287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f13482-4e77-450d-91eb-82fb87896d5c admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:31.161950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a4048a-5a1b-4867-af94-bd0acdc21693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:31.162561 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a4048a-5a1b-4867-af94-bd0acdc21693 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:31.162823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a4048a-5a1b-4867-af94-bd0acdc21693 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:31.163088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a4048a-5a1b-4867-af94-bd0acdc21693 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:31.163721 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a4048a-5a1b-4867-af94-bd0acdc21693 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:31.166030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a4048a-5a1b-4867-af94-bd0acdc21693 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:31.177986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1188/2384] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:32:31 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:32:31.182239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd8d6c1-ebdc-4416-a218-75133411a444 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:31.193363 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd8d6c1-ebdc-4416-a218-75133411a444 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:31.198272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a013af6-e79f-4fd2-b350-457b25c5dd75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:31.198959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a013af6-e79f-4fd2-b350-457b25c5dd75 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:31.199232 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a013af6-e79f-4fd2-b350-457b25c5dd75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:31.199499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a013af6-e79f-4fd2-b350-457b25c5dd75 admin admin] PATH_INFO: `/v3/projects/0ace1a5bdff14a648b45ebd4fa0429d4/users/89eff7eb785648109cddaa69bde8430b/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:31.216058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a013af6-e79f-4fd2-b350-457b25c5dd75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ace1a5bdff14a648b45ebd4fa0429d4, user_id=89eff7eb785648109cddaa69bde8430b, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:31.218908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a013af6-e79f-4fd2-b350-457b25c5dd75 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:31.294737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1197/2385] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:32:31 2022] PUT /identity/v3/projects/0ace1a5bdff14a648b45ebd4fa0429d4/users/89eff7eb785648109cddaa69bde8430b/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:31.306181 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0859dd6a-4415-4e69-b705-52f11deeb5a8 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:31.306476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0859dd6a-4415-4e69-b705-52f11deeb5a8 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:31.306736 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0859dd6a-4415-4e69-b705-52f11deeb5a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:31.334975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-0859dd6a-4415-4e69-b705-52f11deeb5a8 None None] MFA Rules not processed for user `89eff7eb785648109cddaa69bde8430b`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:31.336211 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-0859dd6a-4415-4e69-b705-52f11deeb5a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:31.402251 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1189/2386] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:31 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:31.413434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a013af6-e79f-4fd2-b350-457b25c5dd75 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:31.415272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-8a013af6-e79f-4fd2-b350-457b25c5dd75 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:31.488938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a013af6-e79f-4fd2-b350-457b25c5dd75 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:31.494532 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ee7a42-b83c-4f12-a4d0-446305f97243 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:31.495134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ee7a42-b83c-4f12-a4d0-446305f97243 service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:31.495410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ee7a42-b83c-4f12-a4d0-446305f97243 service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:31.495680 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ee7a42-b83c-4f12-a4d0-446305f97243 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:31.503751 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ee7a42-b83c-4f12-a4d0-446305f97243 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:31.505370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ee7a42-b83c-4f12-a4d0-446305f97243 service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:31.514989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1198/2387] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:32:31 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:32:34.809320 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df6e839-6c3e-4d17-b31c-a4940000e60f None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:34.809991 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df6e839-6c3e-4d17-b31c-a4940000e60f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:34.809991 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df6e839-6c3e-4d17-b31c-a4940000e60f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:34.817806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:34.828979 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-5df6e839-6c3e-4d17-b31c-a4940000e60f None None] MFA Rules not processed for user `a91ac0ba7e8c4418930852b0fa184247`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:34.830507 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5df6e839-6c3e-4d17-b31c-a4940000e60f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:34.878029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ee7a42-b83c-4f12-a4d0-446305f97243 service nova] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:34.888117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1190/2388] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:34 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:34.948763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ee7a42-b83c-4f12-a4d0-446305f97243 service nova] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:34.953486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1813e6a9-9c8e-4f93-92ef-c6b00d02b396 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:34.954158 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1813e6a9-9c8e-4f93-92ef-c6b00d02b396 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:34.958742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1813e6a9-9c8e-4f93-92ef-c6b00d02b396 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:34.958742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1813e6a9-9c8e-4f93-92ef-c6b00d02b396 admin admin] PATH_INFO: `/v3/users/89eff7eb785648109cddaa69bde8430b` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:34.958742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1813e6a9-9c8e-4f93-92ef-c6b00d02b396 admin admin] RBAC: Authorizing `identity:delete_user(user_id=89eff7eb785648109cddaa69bde8430b)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:34.958742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1813e6a9-9c8e-4f93-92ef-c6b00d02b396 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:35.167929 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:35.177973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-1813e6a9-9c8e-4f93-92ef-c6b00d02b396 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89eff7eb785648109cddaa69bde8430b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:35.186636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-1813e6a9-9c8e-4f93-92ef-c6b00d02b396 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89eff7eb785648109cddaa69bde8430b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:35.202978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-1813e6a9-9c8e-4f93-92ef-c6b00d02b396 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89eff7eb785648109cddaa69bde8430b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:35.265174 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-1813e6a9-9c8e-4f93-92ef-c6b00d02b396 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89eff7eb785648109cddaa69bde8430b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:35.265803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-1813e6a9-9c8e-4f93-92ef-c6b00d02b396 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89eff7eb785648109cddaa69bde8430b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:35.272420 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1199/2389] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:32:34 2022] DELETE /identity/v3/users/89eff7eb785648109cddaa69bde8430b => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:32:35.276891 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4c81c4-604a-4388-a076-93771736be4d None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:35.277175 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4c81c4-604a-4388-a076-93771736be4d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:35.277446 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4c81c4-604a-4388-a076-93771736be4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:35.298130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-ca4c81c4-604a-4388-a076-93771736be4d None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:35.299388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ca4c81c4-604a-4388-a076-93771736be4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:35.365335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1191/2390] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:35.375882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1813e6a9-9c8e-4f93-92ef-c6b00d02b396 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:35.377409 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1813e6a9-9c8e-4f93-92ef-c6b00d02b396 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:35.436886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1813e6a9-9c8e-4f93-92ef-c6b00d02b396 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:35.440593 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fed851-d2bb-4929-b091-6da72d260014 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:35.441198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fed851-d2bb-4929-b091-6da72d260014 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:35.441476 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fed851-d2bb-4929-b091-6da72d260014 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:35.441854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fed851-d2bb-4929-b091-6da72d260014 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:35.451465 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fed851-d2bb-4929-b091-6da72d260014 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:35.453079 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fed851-d2bb-4929-b091-6da72d260014 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:35.463889 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1200/2391] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:32:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:32:35.945369 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4c81c4-604a-4388-a076-93771736be4d None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:35.946878 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ca4c81c4-604a-4388-a076-93771736be4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:35.962584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4c81c4-604a-4388-a076-93771736be4d None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:35.968420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bbbdb6-31e9-471d-b4ec-3a1d4adb9fb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:35.969065 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bbbdb6-31e9-471d-b4ec-3a1d4adb9fb1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:35.969428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bbbdb6-31e9-471d-b4ec-3a1d4adb9fb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:35.969775 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bbbdb6-31e9-471d-b4ec-3a1d4adb9fb1 admin admin] PATH_INFO: `/v3/projects/0ace1a5bdff14a648b45ebd4fa0429d4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:35.971041 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bbbdb6-31e9-471d-b4ec-3a1d4adb9fb1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ace1a5bdff14a648b45ebd4fa0429d4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:35.975147 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bbbdb6-31e9-471d-b4ec-3a1d4adb9fb1 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:36.082700 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-b5bbbdb6-31e9-471d-b4ec-3a1d4adb9fb1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ace1a5bdff14a648b45ebd4fa0429d4', 'request_id': 'req-b5bbbdb6-31e9-471d-b4ec-3a1d4adb9fb1'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:36.089156 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-b5bbbdb6-31e9-471d-b4ec-3a1d4adb9fb1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ace1a5bdff14a648b45ebd4fa0429d4', 'request_id': 'req-b5bbbdb6-31e9-471d-b4ec-3a1d4adb9fb1'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:36.095895 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-b5bbbdb6-31e9-471d-b4ec-3a1d4adb9fb1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ace1a5bdff14a648b45ebd4fa0429d4', 'request_id': 'req-b5bbbdb6-31e9-471d-b4ec-3a1d4adb9fb1'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:36.189662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-b5bbbdb6-31e9-471d-b4ec-3a1d4adb9fb1 admin admin] The token cache is being invalidate because project 0ace1a5bdff14a648b45ebd4fa0429d4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:32:36.189879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-b5bbbdb6-31e9-471d-b4ec-3a1d4adb9fb1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:36.190512 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-b5bbbdb6-31e9-471d-b4ec-3a1d4adb9fb1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:36.192002 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1192/2392] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:32:35 2022] DELETE /identity/v3/projects/0ace1a5bdff14a648b45ebd4fa0429d4 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:36.197013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80f6620-a79a-480f-bd24-79aaa211712e None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:36.197476 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80f6620-a79a-480f-bd24-79aaa211712e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:36.197570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80f6620-a79a-480f-bd24-79aaa211712e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:36.223209 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-a80f6620-a79a-480f-bd24-79aaa211712e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:36.223209 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-a80f6620-a79a-480f-bd24-79aaa211712e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:36.281314 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1201/2393] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:36.289828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb7d705-69e1-4704-82e7-400f0653c07e None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:36.290112 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb7d705-69e1-4704-82e7-400f0653c07e None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:36.290386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb7d705-69e1-4704-82e7-400f0653c07e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:36.311608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-7eb7d705-69e1-4704-82e7-400f0653c07e None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:36.312882 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7eb7d705-69e1-4704-82e7-400f0653c07e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:36.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1193/2394] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:36.327206 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80f6620-a79a-480f-bd24-79aaa211712e None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:36.337364 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80f6620-a79a-480f-bd24-79aaa211712e None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:36.342206 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95097e8a-7804-4a69-b7b5-b62cea4b21c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:36.346547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95097e8a-7804-4a69-b7b5-b62cea4b21c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:36.346547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95097e8a-7804-4a69-b7b5-b62cea4b21c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:36.346547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95097e8a-7804-4a69-b7b5-b62cea4b21c3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:36.346547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95097e8a-7804-4a69-b7b5-b62cea4b21c3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:32:36.346547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95097e8a-7804-4a69-b7b5-b62cea4b21c3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:36.346547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95097e8a-7804-4a69-b7b5-b62cea4b21c3 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:36.359316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1202/2395] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:32:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:32:36.364015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb7d705-69e1-4704-82e7-400f0653c07e None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:36.374078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb7d705-69e1-4704-82e7-400f0653c07e None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:36.378905 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ff2a2e-b8ab-471a-886e-478d568bbce0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:36.379510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ff2a2e-b8ab-471a-886e-478d568bbce0 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:36.379787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ff2a2e-b8ab-471a-886e-478d568bbce0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:36.380055 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ff2a2e-b8ab-471a-886e-478d568bbce0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:36.380705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ff2a2e-b8ab-471a-886e-478d568bbce0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:36.382618 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ff2a2e-b8ab-471a-886e-478d568bbce0 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:36.502270 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1194/2396] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:32:36 2022] POST /identity/v3/projects => generated 363 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:32:36.506568 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95097e8a-7804-4a69-b7b5-b62cea4b21c3 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:36.516920 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95097e8a-7804-4a69-b7b5-b62cea4b21c3 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:36.568242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:36.587178 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd586332-2382-402b-84b3-8e52ebfc28c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:36.587911 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd586332-2382-402b-84b3-8e52ebfc28c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:36.588232 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd586332-2382-402b-84b3-8e52ebfc28c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:36.588544 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd586332-2382-402b-84b3-8e52ebfc28c9 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:36.589176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd586332-2382-402b-84b3-8e52ebfc28c9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:36.591171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd586332-2382-402b-84b3-8e52ebfc28c9 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:36.701274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1203/2397] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:32:36 2022] POST /identity/v3/users => generated 282 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:32:36.705753 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ff2a2e-b8ab-471a-886e-478d568bbce0 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:36.713657 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:36.720991 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ff2a2e-b8ab-471a-886e-478d568bbce0 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:36.724222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bb2d56-9c30-42fe-acd3-90bc682b4414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:36.724915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bb2d56-9c30-42fe-acd3-90bc682b4414 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:36.725180 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bb2d56-9c30-42fe-acd3-90bc682b4414 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:36.725734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bb2d56-9c30-42fe-acd3-90bc682b4414 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:36.726183 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bb2d56-9c30-42fe-acd3-90bc682b4414 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:36.728028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bb2d56-9c30-42fe-acd3-90bc682b4414 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:36.740754 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1195/2398] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:32:36 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:32:36.745497 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd586332-2382-402b-84b3-8e52ebfc28c9 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:36.755991 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd586332-2382-402b-84b3-8e52ebfc28c9 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:36.760802 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d5516c-0083-44fb-8d94-ab83cf037165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:36.761411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d5516c-0083-44fb-8d94-ab83cf037165 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:36.769496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d5516c-0083-44fb-8d94-ab83cf037165 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:36.769496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d5516c-0083-44fb-8d94-ab83cf037165 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:36.769496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d5516c-0083-44fb-8d94-ab83cf037165 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:36.769496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d5516c-0083-44fb-8d94-ab83cf037165 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:36.776544 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1204/2399] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:32:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:32:36.780935 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bb2d56-9c30-42fe-acd3-90bc682b4414 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:36.793076 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bb2d56-9c30-42fe-acd3-90bc682b4414 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:36.798087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a25b92-3329-4b20-9729-4b86dcfae789 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:36.798787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a25b92-3329-4b20-9729-4b86dcfae789 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:36.799070 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a25b92-3329-4b20-9729-4b86dcfae789 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:36.799339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a25b92-3329-4b20-9729-4b86dcfae789 admin admin] PATH_INFO: `/v3/projects/1efbbb4ff869487fba7a2077fbd559a4/users/993aab239cf549f0ad4a09a03495cf49/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:36.816204 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a25b92-3329-4b20-9729-4b86dcfae789 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1efbbb4ff869487fba7a2077fbd559a4, user_id=993aab239cf549f0ad4a09a03495cf49, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:36.819110 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a25b92-3329-4b20-9729-4b86dcfae789 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:36.929280 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1196/2400] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:32:36 2022] PUT /identity/v3/projects/1efbbb4ff869487fba7a2077fbd559a4/users/993aab239cf549f0ad4a09a03495cf49/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:36.941483 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7b4226-4a71-40b9-9ad7-43542b74585f None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:36.941760 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7b4226-4a71-40b9-9ad7-43542b74585f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:36.942032 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7b4226-4a71-40b9-9ad7-43542b74585f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:36.973989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-8f7b4226-4a71-40b9-9ad7-43542b74585f None None] MFA Rules not processed for user `993aab239cf549f0ad4a09a03495cf49`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:36.975382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-8f7b4226-4a71-40b9-9ad7-43542b74585f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:37.050102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1205/2401] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:36 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:37.055632 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a25b92-3329-4b20-9729-4b86dcfae789 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:37.129384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a25b92-3329-4b20-9729-4b86dcfae789 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:37.136563 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394dbdda-57fe-4f38-aec3-a53a798a9931 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:37.137179 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394dbdda-57fe-4f38-aec3-a53a798a9931 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:37.137500 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394dbdda-57fe-4f38-aec3-a53a798a9931 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:37.137910 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394dbdda-57fe-4f38-aec3-a53a798a9931 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:37.138861 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394dbdda-57fe-4f38-aec3-a53a798a9931 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:37.141743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394dbdda-57fe-4f38-aec3-a53a798a9931 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:37.262051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1197/2402] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:32:37 2022] POST /identity/v3/projects => generated 365 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:32:37.269617 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7b4226-4a71-40b9-9ad7-43542b74585f None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:37.282187 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7b4226-4a71-40b9-9ad7-43542b74585f None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:37.349640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac1789a-3303-4092-8151-3e5a2b33c59a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:37.350426 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac1789a-3303-4092-8151-3e5a2b33c59a admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:37.350774 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac1789a-3303-4092-8151-3e5a2b33c59a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:37.351115 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac1789a-3303-4092-8151-3e5a2b33c59a admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:37.351832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac1789a-3303-4092-8151-3e5a2b33c59a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:37.353794 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac1789a-3303-4092-8151-3e5a2b33c59a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:37.481966 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1206/2403] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:32:37 2022] POST /identity/v3/users => generated 289 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:32:37.531095 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394dbdda-57fe-4f38-aec3-a53a798a9931 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:37.541871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394dbdda-57fe-4f38-aec3-a53a798a9931 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:37.547965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c9a525-731e-41d9-adcb-914bdb0407aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:37.548692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c9a525-731e-41d9-adcb-914bdb0407aa admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:37.548986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c9a525-731e-41d9-adcb-914bdb0407aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:37.549255 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c9a525-731e-41d9-adcb-914bdb0407aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:37.549896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c9a525-731e-41d9-adcb-914bdb0407aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:37.551691 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c9a525-731e-41d9-adcb-914bdb0407aa admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:37.564064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1198/2404] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:32:37 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:32:37.569116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac1789a-3303-4092-8151-3e5a2b33c59a admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:37.585152 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac1789a-3303-4092-8151-3e5a2b33c59a admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:37.590776 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdd6c09-fdbe-4fc3-9a25-b93a8ff4a687 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:37.591440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdd6c09-fdbe-4fc3-9a25-b93a8ff4a687 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:37.591710 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdd6c09-fdbe-4fc3-9a25-b93a8ff4a687 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:37.592001 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdd6c09-fdbe-4fc3-9a25-b93a8ff4a687 admin admin] PATH_INFO: `/v3/projects/78bd3348ae1d433191c1f203776fe785/users/bead6a56818a4f6fbefe4140eb7797b2/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:37.609925 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdd6c09-fdbe-4fc3-9a25-b93a8ff4a687 admin admin] RBAC: Authorizing `identity:create_grant(project_id=78bd3348ae1d433191c1f203776fe785, user_id=bead6a56818a4f6fbefe4140eb7797b2, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:37.613454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdd6c09-fdbe-4fc3-9a25-b93a8ff4a687 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:37.700632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1207/2405] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:32:37 2022] PUT /identity/v3/projects/78bd3348ae1d433191c1f203776fe785/users/bead6a56818a4f6fbefe4140eb7797b2/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:37.705127 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c9a525-731e-41d9-adcb-914bdb0407aa admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:37.716231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c9a525-731e-41d9-adcb-914bdb0407aa admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:37.780047 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c177b868-7123-42ca-ac16-ad1da453a344 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:37.780662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c177b868-7123-42ca-ac16-ad1da453a344 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:37.780938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c177b868-7123-42ca-ac16-ad1da453a344 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:37.781261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c177b868-7123-42ca-ac16-ad1da453a344 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:37.781851 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c177b868-7123-42ca-ac16-ad1da453a344 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:37.783439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c177b868-7123-42ca-ac16-ad1da453a344 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:37.793872 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1199/2406] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:32:37 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:32:37.798295 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdd6c09-fdbe-4fc3-9a25-b93a8ff4a687 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:37.813498 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdd6c09-fdbe-4fc3-9a25-b93a8ff4a687 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:37.826441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e085c394-9a47-472b-94aa-ca892f1138c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:37.827278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e085c394-9a47-472b-94aa-ca892f1138c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:37.827633 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e085c394-9a47-472b-94aa-ca892f1138c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:37.827978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e085c394-9a47-472b-94aa-ca892f1138c7 admin admin] PATH_INFO: `/v3/projects/78bd3348ae1d433191c1f203776fe785/users/bead6a56818a4f6fbefe4140eb7797b2/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:37.830412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e085c394-9a47-472b-94aa-ca892f1138c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=78bd3348ae1d433191c1f203776fe785, user_id=bead6a56818a4f6fbefe4140eb7797b2, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:37.833255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e085c394-9a47-472b-94aa-ca892f1138c7 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:37.846427 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1208/2407] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:32:37 2022] PUT /identity/v3/projects/78bd3348ae1d433191c1f203776fe785/users/bead6a56818a4f6fbefe4140eb7797b2/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:37.857573 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fc309e-71db-4b95-b8b0-c63bf1175685 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:37.857857 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fc309e-71db-4b95-b8b0-c63bf1175685 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:37.858133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fc309e-71db-4b95-b8b0-c63bf1175685 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:37.911806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-b4fc309e-71db-4b95-b8b0-c63bf1175685 None None] MFA Rules not processed for user `bead6a56818a4f6fbefe4140eb7797b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:37.911806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b4fc309e-71db-4b95-b8b0-c63bf1175685 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:37.941922 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:37.993383 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1200/2408] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:37 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:38.005900 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e085c394-9a47-472b-94aa-ca892f1138c7 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:38.007756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e085c394-9a47-472b-94aa-ca892f1138c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:38.011952 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:38.116046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e085c394-9a47-472b-94aa-ca892f1138c7 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:38.121078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0414d6-94d2-40b2-a807-d958bb768e3c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:38.121755 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0414d6-94d2-40b2-a807-d958bb768e3c service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:38.122030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0414d6-94d2-40b2-a807-d958bb768e3c service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:38.122302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0414d6-94d2-40b2-a807-d958bb768e3c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:38.132770 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0414d6-94d2-40b2-a807-d958bb768e3c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:38.134230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0414d6-94d2-40b2-a807-d958bb768e3c service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:38.144671 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1209/2409] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:32:38 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:32:38.453576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fc309e-71db-4b95-b8b0-c63bf1175685 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:38.468297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fc309e-71db-4b95-b8b0-c63bf1175685 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:38.472351 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09f5be6-7e90-429d-b031-7aea78b15cd8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:38.477287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09f5be6-7e90-429d-b031-7aea78b15cd8 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:38.477287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09f5be6-7e90-429d-b031-7aea78b15cd8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:38.477287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09f5be6-7e90-429d-b031-7aea78b15cd8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:38.541229 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09f5be6-7e90-429d-b031-7aea78b15cd8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:38.543451 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09f5be6-7e90-429d-b031-7aea78b15cd8 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:38.553528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1201/2410] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:32:38 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:32:42.239634 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0414d6-94d2-40b2-a807-d958bb768e3c service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:42.303536 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0414d6-94d2-40b2-a807-d958bb768e3c service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:42.308254 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc3bcc4-94c9-4d2b-9715-3c2246aadcd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:42.308892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc3bcc4-94c9-4d2b-9715-3c2246aadcd2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:42.309171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc3bcc4-94c9-4d2b-9715-3c2246aadcd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:42.309505 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc3bcc4-94c9-4d2b-9715-3c2246aadcd2 admin admin] PATH_INFO: `/v3/users/993aab239cf549f0ad4a09a03495cf49` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:42.310771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc3bcc4-94c9-4d2b-9715-3c2246aadcd2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=993aab239cf549f0ad4a09a03495cf49)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:42.312892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc3bcc4-94c9-4d2b-9715-3c2246aadcd2 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:42.486358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-abc3bcc4-94c9-4d2b-9715-3c2246aadcd2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '993aab239cf549f0ad4a09a03495cf49'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:42.493921 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-abc3bcc4-94c9-4d2b-9715-3c2246aadcd2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '993aab239cf549f0ad4a09a03495cf49'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:42.506471 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-abc3bcc4-94c9-4d2b-9715-3c2246aadcd2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '993aab239cf549f0ad4a09a03495cf49'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:42.579510 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-abc3bcc4-94c9-4d2b-9715-3c2246aadcd2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '993aab239cf549f0ad4a09a03495cf49'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:42.580336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-abc3bcc4-94c9-4d2b-9715-3c2246aadcd2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '993aab239cf549f0ad4a09a03495cf49'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:42.587154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1210/2411] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:32:42 2022] DELETE /identity/v3/users/993aab239cf549f0ad4a09a03495cf49 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:42.592534 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c8325a-cbe5-4037-986e-f1be5ee9cfa6 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:42.593016 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c8325a-cbe5-4037-986e-f1be5ee9cfa6 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:42.593131 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c8325a-cbe5-4037-986e-f1be5ee9cfa6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:42.611800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-03c8325a-cbe5-4037-986e-f1be5ee9cfa6 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:42.613039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-03c8325a-cbe5-4037-986e-f1be5ee9cfa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:42.672819 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1202/2412] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:32:42.683171 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc3bcc4-94c9-4d2b-9715-3c2246aadcd2 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:42.684688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-abc3bcc4-94c9-4d2b-9715-3c2246aadcd2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:42.691177 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:42.744982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc3bcc4-94c9-4d2b-9715-3c2246aadcd2 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:42.748986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ae8082-c179-43fb-b3e9-5eb6a3422929 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:42.749679 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ae8082-c179-43fb-b3e9-5eb6a3422929 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:42.750259 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ae8082-c179-43fb-b3e9-5eb6a3422929 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:42.750879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ae8082-c179-43fb-b3e9-5eb6a3422929 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:42.761351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ae8082-c179-43fb-b3e9-5eb6a3422929 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:42.766127 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ae8082-c179-43fb-b3e9-5eb6a3422929 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:42.772917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1211/2413] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:32:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:32:43.192716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c8325a-cbe5-4037-986e-f1be5ee9cfa6 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:43.194259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-03c8325a-cbe5-4037-986e-f1be5ee9cfa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:43.207940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c8325a-cbe5-4037-986e-f1be5ee9cfa6 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:43.212918 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:43.213599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:43.214029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:43.214309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843 admin admin] PATH_INFO: `/v3/projects/1efbbb4ff869487fba7a2077fbd559a4` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:43.215465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1efbbb4ff869487fba7a2077fbd559a4)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:43.217671 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:43.367734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1efbbb4ff869487fba7a2077fbd559a4', 'request_id': 'req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:43.374027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1efbbb4ff869487fba7a2077fbd559a4', 'request_id': 'req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:43.375111 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:43.380775 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1efbbb4ff869487fba7a2077fbd559a4', 'request_id': 'req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:43.488274 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843 admin admin] The token cache is being invalidate because project 1efbbb4ff869487fba7a2077fbd559a4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:32:43.491962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:43.491962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:43.491962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1203/2414] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:32:43 2022] DELETE /identity/v3/projects/1efbbb4ff869487fba7a2077fbd559a4 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:43.494869 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ae8082-c179-43fb-b3e9-5eb6a3422929 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:43.505561 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ae8082-c179-43fb-b3e9-5eb6a3422929 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:43.565447 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7be280-2064-413c-b082-48df794b7750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:43.566364 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7be280-2064-413c-b082-48df794b7750 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:43.566730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7be280-2064-413c-b082-48df794b7750 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:43.567067 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7be280-2064-413c-b082-48df794b7750 admin admin] PATH_INFO: `/v3/users/bead6a56818a4f6fbefe4140eb7797b2` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:43.568230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7be280-2064-413c-b082-48df794b7750 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bead6a56818a4f6fbefe4140eb7797b2)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:43.570160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7be280-2064-413c-b082-48df794b7750 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:43.840145 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3b7be280-2064-413c-b082-48df794b7750 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bead6a56818a4f6fbefe4140eb7797b2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:43.847067 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3b7be280-2064-413c-b082-48df794b7750 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bead6a56818a4f6fbefe4140eb7797b2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:43.860225 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3b7be280-2064-413c-b082-48df794b7750 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bead6a56818a4f6fbefe4140eb7797b2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:43.928313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3b7be280-2064-413c-b082-48df794b7750 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bead6a56818a4f6fbefe4140eb7797b2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:43.932740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3b7be280-2064-413c-b082-48df794b7750 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bead6a56818a4f6fbefe4140eb7797b2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:43.935632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1212/2415] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:32:43 2022] DELETE /identity/v3/users/bead6a56818a4f6fbefe4140eb7797b2 => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:44.388090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:44.389605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:44.403442 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a687c1-2c4e-4ee4-b8b1-6e3261e01843 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:44.463588 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ec4d2a-3907-4268-b027-809af01328cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:44.464252 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ec4d2a-3907-4268-b027-809af01328cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:44.464530 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ec4d2a-3907-4268-b027-809af01328cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:44.464796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ec4d2a-3907-4268-b027-809af01328cf admin admin] PATH_INFO: `/v3/projects/78bd3348ae1d433191c1f203776fe785` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:44.465890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ec4d2a-3907-4268-b027-809af01328cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=78bd3348ae1d433191c1f203776fe785)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:44.467938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ec4d2a-3907-4268-b027-809af01328cf admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:44.620611 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e8ec4d2a-3907-4268-b027-809af01328cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78bd3348ae1d433191c1f203776fe785', 'request_id': 'req-e8ec4d2a-3907-4268-b027-809af01328cf'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:44.626032 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e8ec4d2a-3907-4268-b027-809af01328cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78bd3348ae1d433191c1f203776fe785', 'request_id': 'req-e8ec4d2a-3907-4268-b027-809af01328cf'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:44.632198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e8ec4d2a-3907-4268-b027-809af01328cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78bd3348ae1d433191c1f203776fe785', 'request_id': 'req-e8ec4d2a-3907-4268-b027-809af01328cf'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:44.732847 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e8ec4d2a-3907-4268-b027-809af01328cf admin admin] The token cache is being invalidate because project 78bd3348ae1d433191c1f203776fe785 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:32:44.733162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e8ec4d2a-3907-4268-b027-809af01328cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:44.733796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-e8ec4d2a-3907-4268-b027-809af01328cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:44.735309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1204/2416] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:32:44 2022] DELETE /identity/v3/projects/78bd3348ae1d433191c1f203776fe785 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:44.740212 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f4a4e6-60df-421a-8e5f-ebf3cd6775c0 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:44.740706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f4a4e6-60df-421a-8e5f-ebf3cd6775c0 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:44.740876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f4a4e6-60df-421a-8e5f-ebf3cd6775c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:44.761432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-32f4a4e6-60df-421a-8e5f-ebf3cd6775c0 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:44.762628 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-32f4a4e6-60df-421a-8e5f-ebf3cd6775c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:44.782683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:44.822356 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1213/2417] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:44.830731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a537eca5-0d72-4258-be8c-53c31b031da9 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:44.831012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a537eca5-0d72-4258-be8c-53c31b031da9 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:44.831283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a537eca5-0d72-4258-be8c-53c31b031da9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:44.851430 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-a537eca5-0d72-4258-be8c-53c31b031da9 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:44.852827 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a537eca5-0d72-4258-be8c-53c31b031da9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:44.861327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1205/2418] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:44.865613 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f4a4e6-60df-421a-8e5f-ebf3cd6775c0 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:44.875958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f4a4e6-60df-421a-8e5f-ebf3cd6775c0 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:44.880519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b18a2ae-8cc4-4305-8df7-7190ceee1d73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:44.881167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18a2ae-8cc4-4305-8df7-7190ceee1d73 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:44.881438 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18a2ae-8cc4-4305-8df7-7190ceee1d73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:44.881707 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18a2ae-8cc4-4305-8df7-7190ceee1d73 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:44.882352 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b18a2ae-8cc4-4305-8df7-7190ceee1d73 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:32:44.882650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b18a2ae-8cc4-4305-8df7-7190ceee1d73 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:44.884282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b18a2ae-8cc4-4305-8df7-7190ceee1d73 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:44.895750 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1214/2419] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:32:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:32:44.900261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a537eca5-0d72-4258-be8c-53c31b031da9 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:44.909930 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a537eca5-0d72-4258-be8c-53c31b031da9 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:44.914663 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87e3892-ca90-4a54-9a6a-de6033e66409 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:44.915286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87e3892-ca90-4a54-9a6a-de6033e66409 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:44.915609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87e3892-ca90-4a54-9a6a-de6033e66409 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:44.915879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87e3892-ca90-4a54-9a6a-de6033e66409 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:44.917035 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87e3892-ca90-4a54-9a6a-de6033e66409 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:44.919002 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87e3892-ca90-4a54-9a6a-de6033e66409 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:45.061772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1206/2420] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:32:44 2022] POST /identity/v3/projects => generated 377 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:32:45.065938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b18a2ae-8cc4-4305-8df7-7190ceee1d73 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:45.076009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b18a2ae-8cc4-4305-8df7-7190ceee1d73 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:45.145020 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc045eb7-8b9f-42b0-8d6a-c428e528ebbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:45.145537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc045eb7-8b9f-42b0-8d6a-c428e528ebbf admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:45.145810 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc045eb7-8b9f-42b0-8d6a-c428e528ebbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:45.146078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc045eb7-8b9f-42b0-8d6a-c428e528ebbf admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:45.146711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc045eb7-8b9f-42b0-8d6a-c428e528ebbf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:45.148672 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc045eb7-8b9f-42b0-8d6a-c428e528ebbf admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:45.288923 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1215/2421] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:32:45 2022] POST /identity/v3/users => generated 289 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:32:45.295502 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87e3892-ca90-4a54-9a6a-de6033e66409 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:45.306290 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87e3892-ca90-4a54-9a6a-de6033e66409 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:45.310413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a75478-cd69-45c2-b932-2ccbcf1e6c96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:45.311008 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a75478-cd69-45c2-b932-2ccbcf1e6c96 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:45.311277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a75478-cd69-45c2-b932-2ccbcf1e6c96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:45.311542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a75478-cd69-45c2-b932-2ccbcf1e6c96 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:45.312271 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a75478-cd69-45c2-b932-2ccbcf1e6c96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:45.313792 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a75478-cd69-45c2-b932-2ccbcf1e6c96 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:45.325263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1207/2422] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:32:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:32:45.329554 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc045eb7-8b9f-42b0-8d6a-c428e528ebbf admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:45.340717 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc045eb7-8b9f-42b0-8d6a-c428e528ebbf admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:45.345855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab65a7bd-c5e8-4a3f-953e-25d980cedb35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:45.346472 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab65a7bd-c5e8-4a3f-953e-25d980cedb35 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:45.346749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab65a7bd-c5e8-4a3f-953e-25d980cedb35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:45.347020 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab65a7bd-c5e8-4a3f-953e-25d980cedb35 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:45.347666 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab65a7bd-c5e8-4a3f-953e-25d980cedb35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:45.349312 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab65a7bd-c5e8-4a3f-953e-25d980cedb35 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:45.362307 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1216/2423] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:32:45 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:32:45.366513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a75478-cd69-45c2-b932-2ccbcf1e6c96 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:45.378594 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a75478-cd69-45c2-b932-2ccbcf1e6c96 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:45.383221 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d45b23b-42b2-44bb-830d-c2eae37e2965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:45.383912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d45b23b-42b2-44bb-830d-c2eae37e2965 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:45.384443 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d45b23b-42b2-44bb-830d-c2eae37e2965 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:45.384719 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d45b23b-42b2-44bb-830d-c2eae37e2965 admin admin] PATH_INFO: `/v3/projects/44390bed1802429c92973c924ce98407/users/f82f879dd7854e3ab434d1cb229af87c/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:45.399816 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d45b23b-42b2-44bb-830d-c2eae37e2965 admin admin] RBAC: Authorizing `identity:create_grant(project_id=44390bed1802429c92973c924ce98407, user_id=f82f879dd7854e3ab434d1cb229af87c, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:45.403078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d45b23b-42b2-44bb-830d-c2eae37e2965 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:45.534739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1208/2424] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:32:45 2022] PUT /identity/v3/projects/44390bed1802429c92973c924ce98407/users/f82f879dd7854e3ab434d1cb229af87c/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:45.546508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29576e1a-7e51-4bcd-af72-8a6d87554783 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:45.546789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29576e1a-7e51-4bcd-af72-8a6d87554783 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:45.547063 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29576e1a-7e51-4bcd-af72-8a6d87554783 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:45.577674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-29576e1a-7e51-4bcd-af72-8a6d87554783 None None] MFA Rules not processed for user `f82f879dd7854e3ab434d1cb229af87c`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:45.579076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-29576e1a-7e51-4bcd-af72-8a6d87554783 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:45.653242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1217/2425] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:45 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:45.657758 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d45b23b-42b2-44bb-830d-c2eae37e2965 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:45.674090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:45.725083 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d45b23b-42b2-44bb-830d-c2eae37e2965 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:45.731785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2329306-d21b-4c45-8baa-1f9df0454632 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:45.732452 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2329306-d21b-4c45-8baa-1f9df0454632 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:45.732723 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2329306-d21b-4c45-8baa-1f9df0454632 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:45.732991 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2329306-d21b-4c45-8baa-1f9df0454632 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:45.733627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2329306-d21b-4c45-8baa-1f9df0454632 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:45.735530 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2329306-d21b-4c45-8baa-1f9df0454632 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:45.850889 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1209/2426] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:32:45 2022] POST /identity/v3/projects => generated 379 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:32:45.855433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29576e1a-7e51-4bcd-af72-8a6d87554783 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:45.867086 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29576e1a-7e51-4bcd-af72-8a6d87554783 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:45.889789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:45.929341 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff08527-5798-4c86-bcec-65b69a96b34f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:45.929947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff08527-5798-4c86-bcec-65b69a96b34f admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:45.930223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff08527-5798-4c86-bcec-65b69a96b34f admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:45.930491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff08527-5798-4c86-bcec-65b69a96b34f admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:45.931116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff08527-5798-4c86-bcec-65b69a96b34f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:45.933682 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff08527-5798-4c86-bcec-65b69a96b34f admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:46.030050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1218/2427] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:32:45 2022] POST /identity/v3/users => generated 296 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:32:46.034000 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2329306-d21b-4c45-8baa-1f9df0454632 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:46.044581 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2329306-d21b-4c45-8baa-1f9df0454632 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:46.049214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52393b84-2459-4adb-a34b-67d2d05abd67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:46.049827 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52393b84-2459-4adb-a34b-67d2d05abd67 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:46.050077 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52393b84-2459-4adb-a34b-67d2d05abd67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:46.050344 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52393b84-2459-4adb-a34b-67d2d05abd67 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:46.050972 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52393b84-2459-4adb-a34b-67d2d05abd67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:46.052623 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52393b84-2459-4adb-a34b-67d2d05abd67 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:46.063569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1210/2428] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:32:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:32:46.070380 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff08527-5798-4c86-bcec-65b69a96b34f admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:46.083534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff08527-5798-4c86-bcec-65b69a96b34f admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:46.089170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d7ed4f-d9cf-4f41-b2e5-88c72790f4aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:46.089840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d7ed4f-d9cf-4f41-b2e5-88c72790f4aa admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:46.090116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d7ed4f-d9cf-4f41-b2e5-88c72790f4aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:46.090391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d7ed4f-d9cf-4f41-b2e5-88c72790f4aa admin admin] PATH_INFO: `/v3/projects/b1dc72f9a260426fb59d4691c8dd8d15/users/d46708dfb455405199c81e911f2ae2d7/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:46.105968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d7ed4f-d9cf-4f41-b2e5-88c72790f4aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1dc72f9a260426fb59d4691c8dd8d15, user_id=d46708dfb455405199c81e911f2ae2d7, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:46.108859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d7ed4f-d9cf-4f41-b2e5-88c72790f4aa admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:46.253164 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1219/2429] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:32:46 2022] PUT /identity/v3/projects/b1dc72f9a260426fb59d4691c8dd8d15/users/d46708dfb455405199c81e911f2ae2d7/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:46.258460 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52393b84-2459-4adb-a34b-67d2d05abd67 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:46.269247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52393b84-2459-4adb-a34b-67d2d05abd67 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:46.332134 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc93ed58-e15c-4e10-92e1-69cfdd1415c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:46.333316 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc93ed58-e15c-4e10-92e1-69cfdd1415c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:46.333316 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc93ed58-e15c-4e10-92e1-69cfdd1415c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:46.333519 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc93ed58-e15c-4e10-92e1-69cfdd1415c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:46.334073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc93ed58-e15c-4e10-92e1-69cfdd1415c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:46.335730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc93ed58-e15c-4e10-92e1-69cfdd1415c5 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:46.347766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1211/2430] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:32:46 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:32:46.352185 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d7ed4f-d9cf-4f41-b2e5-88c72790f4aa admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:46.363340 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d7ed4f-d9cf-4f41-b2e5-88c72790f4aa admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:46.368470 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67107868-bb0a-4f17-a38a-cffc18adb473 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:46.369162 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67107868-bb0a-4f17-a38a-cffc18adb473 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:46.369436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67107868-bb0a-4f17-a38a-cffc18adb473 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:46.370301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67107868-bb0a-4f17-a38a-cffc18adb473 admin admin] PATH_INFO: `/v3/projects/b1dc72f9a260426fb59d4691c8dd8d15/users/d46708dfb455405199c81e911f2ae2d7/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:46.372250 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67107868-bb0a-4f17-a38a-cffc18adb473 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1dc72f9a260426fb59d4691c8dd8d15, user_id=d46708dfb455405199c81e911f2ae2d7, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:46.374951 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67107868-bb0a-4f17-a38a-cffc18adb473 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:46.385973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1220/2431] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:32:46 2022] PUT /identity/v3/projects/b1dc72f9a260426fb59d4691c8dd8d15/users/d46708dfb455405199c81e911f2ae2d7/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:46.396396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797751a1-952e-43df-87dc-40ebf4bb4a17 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:46.396682 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797751a1-952e-43df-87dc-40ebf4bb4a17 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:46.396952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797751a1-952e-43df-87dc-40ebf4bb4a17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:46.427047 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-797751a1-952e-43df-87dc-40ebf4bb4a17 None None] MFA Rules not processed for user `d46708dfb455405199c81e911f2ae2d7`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:46.428429 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-797751a1-952e-43df-87dc-40ebf4bb4a17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:46.497063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1212/2432] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:46 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:46.509765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67107868-bb0a-4f17-a38a-cffc18adb473 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:46.511551 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-67107868-bb0a-4f17-a38a-cffc18adb473 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:46.569895 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67107868-bb0a-4f17-a38a-cffc18adb473 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:46.573902 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e008cb1d-10fa-434b-bf5f-01a1b7d1cc77 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:46.574500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e008cb1d-10fa-434b-bf5f-01a1b7d1cc77 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:46.574772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e008cb1d-10fa-434b-bf5f-01a1b7d1cc77 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:46.575037 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e008cb1d-10fa-434b-bf5f-01a1b7d1cc77 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:46.584749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e008cb1d-10fa-434b-bf5f-01a1b7d1cc77 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:46.586160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e008cb1d-10fa-434b-bf5f-01a1b7d1cc77 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:46.596248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1221/2433] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:32:46 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:32:47.745418 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797751a1-952e-43df-87dc-40ebf4bb4a17 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:47.755420 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797751a1-952e-43df-87dc-40ebf4bb4a17 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:47.760011 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b8f3bc-b4aa-46c0-9e4b-8d678da5afa3 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd46708dfb455405199c81e911f2ae2d7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1dc72f9a260426fb59d4691c8dd8d15', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:47.764887 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b8f3bc-b4aa-46c0-9e4b-8d678da5afa3 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:47.764887 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b8f3bc-b4aa-46c0-9e4b-8d678da5afa3 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:47.764887 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b8f3bc-b4aa-46c0-9e4b-8d678da5afa3 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:47.764887 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b8f3bc-b4aa-46c0-9e4b-8d678da5afa3 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:32:47.764887 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b8f3bc-b4aa-46c0-9e4b-8d678da5afa3 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:47.765376 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b8f3bc-b4aa-46c0-9e4b-8d678da5afa3 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:47.766206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:47.776128 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1213/2434] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:32:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:32:47.780391 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e008cb1d-10fa-434b-bf5f-01a1b7d1cc77 service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:47.789513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e008cb1d-10fa-434b-bf5f-01a1b7d1cc77 service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:47.794720 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05d6177-0f42-42fb-8445-50ef92d9ec25 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd46708dfb455405199c81e911f2ae2d7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1dc72f9a260426fb59d4691c8dd8d15', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:47.799197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05d6177-0f42-42fb-8445-50ef92d9ec25 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:47.799197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05d6177-0f42-42fb-8445-50ef92d9ec25 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:47.799197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05d6177-0f42-42fb-8445-50ef92d9ec25 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:47.799197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05d6177-0f42-42fb-8445-50ef92d9ec25 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:47.799197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05d6177-0f42-42fb-8445-50ef92d9ec25 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:47.882799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1222/2435] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:32:47 2022] POST /identity/v3/projects => generated 355 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:32:47.887068 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b8f3bc-b4aa-46c0-9e4b-8d678da5afa3 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:47.897057 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b8f3bc-b4aa-46c0-9e4b-8d678da5afa3 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:47.955055 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d834c8f4-6fa9-4b6b-8a27-36d06f86e7ec tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd46708dfb455405199c81e911f2ae2d7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1dc72f9a260426fb59d4691c8dd8d15', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:47.955629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d834c8f4-6fa9-4b6b-8a27-36d06f86e7ec tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:47.955905 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d834c8f4-6fa9-4b6b-8a27-36d06f86e7ec tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:47.956210 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d834c8f4-6fa9-4b6b-8a27-36d06f86e7ec tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:47.956846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d834c8f4-6fa9-4b6b-8a27-36d06f86e7ec tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:32:47.957145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d834c8f4-6fa9-4b6b-8a27-36d06f86e7ec tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:47.958720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d834c8f4-6fa9-4b6b-8a27-36d06f86e7ec tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:47.970977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1214/2436] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:32:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:32:48.010678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05d6177-0f42-42fb-8445-50ef92d9ec25 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:48.021383 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05d6177-0f42-42fb-8445-50ef92d9ec25 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:48.026368 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4319f56-1620-4532-bae0-6978c2d3c783 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd46708dfb455405199c81e911f2ae2d7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1dc72f9a260426fb59d4691c8dd8d15', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:48.026994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4319f56-1620-4532-bae0-6978c2d3c783 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:48.027271 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4319f56-1620-4532-bae0-6978c2d3c783 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:48.027539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4319f56-1620-4532-bae0-6978c2d3c783 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] PATH_INFO: `/v3/projects/ffde3cdbda1c48c98867d130070d353e` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:48.028679 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4319f56-1620-4532-bae0-6978c2d3c783 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] RBAC: Authorizing `identity:delete_project(project_id=ffde3cdbda1c48c98867d130070d353e)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:48.030564 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4319f56-1620-4532-bae0-6978c2d3c783 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:48.134275 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e4319f56-1620-4532-bae0-6978c2d3c783 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ffde3cdbda1c48c98867d130070d353e', 'request_id': 'req-e4319f56-1620-4532-bae0-6978c2d3c783'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:48.135972 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=90,keystone:INSERT=6,keystone:DELETE=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:48.223734 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e4319f56-1620-4532-bae0-6978c2d3c783 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ffde3cdbda1c48c98867d130070d353e', 'request_id': 'req-e4319f56-1620-4532-bae0-6978c2d3c783'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:48.229757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e4319f56-1620-4532-bae0-6978c2d3c783 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ffde3cdbda1c48c98867d130070d353e', 'request_id': 'req-e4319f56-1620-4532-bae0-6978c2d3c783'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:48.241937 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e4319f56-1620-4532-bae0-6978c2d3c783 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] The token cache is being invalidate because project ffde3cdbda1c48c98867d130070d353e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:32:48.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e4319f56-1620-4532-bae0-6978c2d3c783 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:48.242873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e4319f56-1620-4532-bae0-6978c2d3c783 tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:48.246800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1223/2437] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:32:48 2022] DELETE /identity/v3/projects/ffde3cdbda1c48c98867d130070d353e => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:48.884178 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d834c8f4-6fa9-4b6b-8a27-36d06f86e7ec tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:48.888840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d834c8f4-6fa9-4b6b-8a27-36d06f86e7ec tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:48.968928 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d834c8f4-6fa9-4b6b-8a27-36d06f86e7ec tempest-VolumeQuotaClassesTest-846779125 tempest-VolumeQuotaClassesTest-846779125-project-admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:49.040896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee195265-5d85-4f3e-b50b-33a1f1b3bc6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:49.041258 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee195265-5d85-4f3e-b50b-33a1f1b3bc6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:49.041529 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee195265-5d85-4f3e-b50b-33a1f1b3bc6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:49.041795 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee195265-5d85-4f3e-b50b-33a1f1b3bc6a admin admin] PATH_INFO: `/v3/users/f82f879dd7854e3ab434d1cb229af87c` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:49.042932 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee195265-5d85-4f3e-b50b-33a1f1b3bc6a admin admin] RBAC: Authorizing `identity:delete_user(user_id=f82f879dd7854e3ab434d1cb229af87c)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:49.045008 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee195265-5d85-4f3e-b50b-33a1f1b3bc6a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:49.279824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:49.285726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ee195265-5d85-4f3e-b50b-33a1f1b3bc6a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f82f879dd7854e3ab434d1cb229af87c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:49.293090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ee195265-5d85-4f3e-b50b-33a1f1b3bc6a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f82f879dd7854e3ab434d1cb229af87c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:49.294316 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ee195265-5d85-4f3e-b50b-33a1f1b3bc6a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f82f879dd7854e3ab434d1cb229af87c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:49.306361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ee195265-5d85-4f3e-b50b-33a1f1b3bc6a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f82f879dd7854e3ab434d1cb229af87c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:49.383657 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ee195265-5d85-4f3e-b50b-33a1f1b3bc6a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f82f879dd7854e3ab434d1cb229af87c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:49.390539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1215/2438] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:32:48 2022] DELETE /identity/v3/users/f82f879dd7854e3ab434d1cb229af87c => generated 0 bytes in 508 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:49.395165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de3af3a-4b12-4627-b63d-bd313d06d142 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:49.395675 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de3af3a-4b12-4627-b63d-bd313d06d142 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:49.395776 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de3af3a-4b12-4627-b63d-bd313d06d142 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:49.414523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-9de3af3a-4b12-4627-b63d-bd313d06d142 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:49.416203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9de3af3a-4b12-4627-b63d-bd313d06d142 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:49.476566 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1224/2439] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:49.486847 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee195265-5d85-4f3e-b50b-33a1f1b3bc6a admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:49.488293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ee195265-5d85-4f3e-b50b-33a1f1b3bc6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:49.548157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee195265-5d85-4f3e-b50b-33a1f1b3bc6a admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:49.551753 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e64242b-ffdc-43e2-8c57-341fbd5a61e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:49.552387 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e64242b-ffdc-43e2-8c57-341fbd5a61e0 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:49.552665 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e64242b-ffdc-43e2-8c57-341fbd5a61e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:49.552931 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e64242b-ffdc-43e2-8c57-341fbd5a61e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:49.562962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e64242b-ffdc-43e2-8c57-341fbd5a61e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:49.564424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e64242b-ffdc-43e2-8c57-341fbd5a61e0 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:49.574157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1216/2440] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:32:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:32:50.117014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de3af3a-4b12-4627-b63d-bd313d06d142 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:50.119095 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9de3af3a-4b12-4627-b63d-bd313d06d142 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:50.133582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de3af3a-4b12-4627-b63d-bd313d06d142 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:50.139634 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04618f2-730c-4583-b46a-ec186bbc0cdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:50.140636 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04618f2-730c-4583-b46a-ec186bbc0cdd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:50.140761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04618f2-730c-4583-b46a-ec186bbc0cdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:50.140990 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04618f2-730c-4583-b46a-ec186bbc0cdd admin admin] PATH_INFO: `/v3/projects/44390bed1802429c92973c924ce98407` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:50.142204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04618f2-730c-4583-b46a-ec186bbc0cdd admin admin] RBAC: Authorizing `identity:delete_project(project_id=44390bed1802429c92973c924ce98407)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:50.144173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04618f2-730c-4583-b46a-ec186bbc0cdd admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:50.303757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b04618f2-730c-4583-b46a-ec186bbc0cdd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '44390bed1802429c92973c924ce98407', 'request_id': 'req-b04618f2-730c-4583-b46a-ec186bbc0cdd'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:50.408155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b04618f2-730c-4583-b46a-ec186bbc0cdd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '44390bed1802429c92973c924ce98407', 'request_id': 'req-b04618f2-730c-4583-b46a-ec186bbc0cdd'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:50.414514 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b04618f2-730c-4583-b46a-ec186bbc0cdd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '44390bed1802429c92973c924ce98407', 'request_id': 'req-b04618f2-730c-4583-b46a-ec186bbc0cdd'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:50.426706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b04618f2-730c-4583-b46a-ec186bbc0cdd admin admin] The token cache is being invalidate because project 44390bed1802429c92973c924ce98407 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:32:50.432328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b04618f2-730c-4583-b46a-ec186bbc0cdd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:50.432328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b04618f2-730c-4583-b46a-ec186bbc0cdd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:50.432328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1225/2441] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:32:50 2022] DELETE /identity/v3/projects/44390bed1802429c92973c924ce98407 => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:50.433621 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e64242b-ffdc-43e2-8c57-341fbd5a61e0 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:50.444473 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e64242b-ffdc-43e2-8c57-341fbd5a61e0 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:50.522984 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc14697-7ca2-46a5-a2a9-bf49dcfd008f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:50.523811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc14697-7ca2-46a5-a2a9-bf49dcfd008f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:50.524178 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc14697-7ca2-46a5-a2a9-bf49dcfd008f admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:50.524371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc14697-7ca2-46a5-a2a9-bf49dcfd008f admin admin] PATH_INFO: `/v3/users/d46708dfb455405199c81e911f2ae2d7` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:50.526254 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc14697-7ca2-46a5-a2a9-bf49dcfd008f admin admin] RBAC: Authorizing `identity:delete_user(user_id=d46708dfb455405199c81e911f2ae2d7)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:50.527685 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc14697-7ca2-46a5-a2a9-bf49dcfd008f admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:50.726390 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5fc14697-7ca2-46a5-a2a9-bf49dcfd008f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd46708dfb455405199c81e911f2ae2d7'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:50.735569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5fc14697-7ca2-46a5-a2a9-bf49dcfd008f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd46708dfb455405199c81e911f2ae2d7'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:50.736514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5fc14697-7ca2-46a5-a2a9-bf49dcfd008f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd46708dfb455405199c81e911f2ae2d7'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:50.742731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:50.752182 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5fc14697-7ca2-46a5-a2a9-bf49dcfd008f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd46708dfb455405199c81e911f2ae2d7'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:50.837396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5fc14697-7ca2-46a5-a2a9-bf49dcfd008f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd46708dfb455405199c81e911f2ae2d7'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:50.844783 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1217/2442] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:32:50 2022] DELETE /identity/v3/users/d46708dfb455405199c81e911f2ae2d7 => generated 0 bytes in 412 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:51.278973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04618f2-730c-4583-b46a-ec186bbc0cdd admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:51.280645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-b04618f2-730c-4583-b46a-ec186bbc0cdd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:51.294556 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04618f2-730c-4583-b46a-ec186bbc0cdd admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:51.354755 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5532a68-682b-44fd-8679-6aef6db5cf20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:51.355394 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5532a68-682b-44fd-8679-6aef6db5cf20 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:51.355670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5532a68-682b-44fd-8679-6aef6db5cf20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:51.355942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5532a68-682b-44fd-8679-6aef6db5cf20 admin admin] PATH_INFO: `/v3/projects/b1dc72f9a260426fb59d4691c8dd8d15` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:51.357775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5532a68-682b-44fd-8679-6aef6db5cf20 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b1dc72f9a260426fb59d4691c8dd8d15)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:51.362535 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5532a68-682b-44fd-8679-6aef6db5cf20 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:51.362535 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:INSERT=2,keystone:DELETE=10,keystone:SELECT=88 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:51.456615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b5532a68-682b-44fd-8679-6aef6db5cf20 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b1dc72f9a260426fb59d4691c8dd8d15', 'request_id': 'req-b5532a68-682b-44fd-8679-6aef6db5cf20'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:51.511771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b5532a68-682b-44fd-8679-6aef6db5cf20 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b1dc72f9a260426fb59d4691c8dd8d15', 'request_id': 'req-b5532a68-682b-44fd-8679-6aef6db5cf20'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:51.520610 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b5532a68-682b-44fd-8679-6aef6db5cf20 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b1dc72f9a260426fb59d4691c8dd8d15', 'request_id': 'req-b5532a68-682b-44fd-8679-6aef6db5cf20'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:51.533507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b5532a68-682b-44fd-8679-6aef6db5cf20 admin admin] The token cache is being invalidate because project b1dc72f9a260426fb59d4691c8dd8d15 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:32:51.533798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b5532a68-682b-44fd-8679-6aef6db5cf20 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:51.534482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b5532a68-682b-44fd-8679-6aef6db5cf20 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:32:51.536008 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1226/2443] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:32:51 2022] DELETE /identity/v3/projects/b1dc72f9a260426fb59d4691c8dd8d15 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:51.541105 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d588475b-db5a-4a2d-916a-575f38ba383b None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:51.541625 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d588475b-db5a-4a2d-916a-575f38ba383b None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:51.541834 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d588475b-db5a-4a2d-916a-575f38ba383b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:51.561920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-d588475b-db5a-4a2d-916a-575f38ba383b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:51.563140 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d588475b-db5a-4a2d-916a-575f38ba383b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:51.631408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1218/2444] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:51.639944 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e72eb44-8a97-43c4-85c0-9a793b794096 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:51.640330 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e72eb44-8a97-43c4-85c0-9a793b794096 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:51.640638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e72eb44-8a97-43c4-85c0-9a793b794096 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:51.661238 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-8e72eb44-8a97-43c4-85c0-9a793b794096 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:51.662405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-8e72eb44-8a97-43c4-85c0-9a793b794096 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:51.670654 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1227/2445] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:51.675175 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d588475b-db5a-4a2d-916a-575f38ba383b None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:51.685154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d588475b-db5a-4a2d-916a-575f38ba383b None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:51.689854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94da565-4f69-47dd-8692-66cf494e4e04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:51.690435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94da565-4f69-47dd-8692-66cf494e4e04 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:51.690706 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94da565-4f69-47dd-8692-66cf494e4e04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:51.690970 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94da565-4f69-47dd-8692-66cf494e4e04 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:51.691609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94da565-4f69-47dd-8692-66cf494e4e04 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:32:51.691904 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94da565-4f69-47dd-8692-66cf494e4e04 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:51.693529 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94da565-4f69-47dd-8692-66cf494e4e04 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:51.706175 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1219/2446] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:32:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:32:51.710590 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e72eb44-8a97-43c4-85c0-9a793b794096 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:51.721518 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e72eb44-8a97-43c4-85c0-9a793b794096 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:51.726274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c59f77-4c52-45ac-a430-ab6817f53c13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:51.726873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c59f77-4c52-45ac-a430-ab6817f53c13 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:51.727150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c59f77-4c52-45ac-a430-ab6817f53c13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:51.727414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c59f77-4c52-45ac-a430-ab6817f53c13 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:51.728041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c59f77-4c52-45ac-a430-ab6817f53c13 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:51.729952 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c59f77-4c52-45ac-a430-ab6817f53c13 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:51.820299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1228/2447] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:32:51 2022] POST /identity/v3/projects => generated 385 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:32:51.824509 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94da565-4f69-47dd-8692-66cf494e4e04 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:51.835566 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94da565-4f69-47dd-8692-66cf494e4e04 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:51.893077 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45ca841-62f8-49e8-9caa-0983ac87bd4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:51.893676 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45ca841-62f8-49e8-9caa-0983ac87bd4a admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:51.893950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45ca841-62f8-49e8-9caa-0983ac87bd4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:51.894244 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45ca841-62f8-49e8-9caa-0983ac87bd4a admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:51.894854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45ca841-62f8-49e8-9caa-0983ac87bd4a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:51.896796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45ca841-62f8-49e8-9caa-0983ac87bd4a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:52.014095 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1220/2448] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:32:51 2022] POST /identity/v3/users => generated 293 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:32:52.018533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c59f77-4c52-45ac-a430-ab6817f53c13 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:52.029276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c59f77-4c52-45ac-a430-ab6817f53c13 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:52.033950 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334ddbb1-f81f-42ec-8ea0-92100d198c82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:52.034568 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334ddbb1-f81f-42ec-8ea0-92100d198c82 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:52.034840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334ddbb1-f81f-42ec-8ea0-92100d198c82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:52.035109 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334ddbb1-f81f-42ec-8ea0-92100d198c82 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:52.035753 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334ddbb1-f81f-42ec-8ea0-92100d198c82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:52.037380 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334ddbb1-f81f-42ec-8ea0-92100d198c82 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:52.049555 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1229/2449] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:32:52 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:32:52.055859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45ca841-62f8-49e8-9caa-0983ac87bd4a admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:52.067222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45ca841-62f8-49e8-9caa-0983ac87bd4a admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:52.078262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e41e8ba-e0a6-4d37-a8b1-6396eeeb3abf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:52.078832 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e41e8ba-e0a6-4d37-a8b1-6396eeeb3abf admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:52.079111 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e41e8ba-e0a6-4d37-a8b1-6396eeeb3abf admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:52.079529 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e41e8ba-e0a6-4d37-a8b1-6396eeeb3abf admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:52.080473 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e41e8ba-e0a6-4d37-a8b1-6396eeeb3abf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:52.082098 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e41e8ba-e0a6-4d37-a8b1-6396eeeb3abf admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:52.093983 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1221/2450] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:32:52 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:32:52.098431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334ddbb1-f81f-42ec-8ea0-92100d198c82 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:52.108921 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334ddbb1-f81f-42ec-8ea0-92100d198c82 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:52.113583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c793a6-4def-48a0-b54d-020a634da4a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:52.114273 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c793a6-4def-48a0-b54d-020a634da4a2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:52.114556 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c793a6-4def-48a0-b54d-020a634da4a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:52.114830 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c793a6-4def-48a0-b54d-020a634da4a2 admin admin] PATH_INFO: `/v3/projects/3a0243a7f1ff44a1bb6fdfda13433961/users/228e7e06d92341cab263dcbf6bc6647e/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:52.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c793a6-4def-48a0-b54d-020a634da4a2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a0243a7f1ff44a1bb6fdfda13433961, user_id=228e7e06d92341cab263dcbf6bc6647e, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:52.133083 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c793a6-4def-48a0-b54d-020a634da4a2 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:52.220535 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1230/2451] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:32:52 2022] PUT /identity/v3/projects/3a0243a7f1ff44a1bb6fdfda13433961/users/228e7e06d92341cab263dcbf6bc6647e/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:52.232455 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8175661-8bc7-4268-b656-a50bc7606331 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:52.232739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8175661-8bc7-4268-b656-a50bc7606331 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:52.233013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8175661-8bc7-4268-b656-a50bc7606331 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:52.262004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-a8175661-8bc7-4268-b656-a50bc7606331 None None] MFA Rules not processed for user `228e7e06d92341cab263dcbf6bc6647e`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:52.263226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a8175661-8bc7-4268-b656-a50bc7606331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:52.296853 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:DELETE=3,keystone:SELECT=93,keystone:INSERT=4 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:52.330708 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1222/2452] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:52 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:52.335922 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c793a6-4def-48a0-b54d-020a634da4a2 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:52.401615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c793a6-4def-48a0-b54d-020a634da4a2 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:52.406423 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbda5eec-e7af-43c8-bab0-a6bd99ee46b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:52.407025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbda5eec-e7af-43c8-bab0-a6bd99ee46b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:52.407299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbda5eec-e7af-43c8-bab0-a6bd99ee46b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:52.407566 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbda5eec-e7af-43c8-bab0-a6bd99ee46b8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:52.408221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbda5eec-e7af-43c8-bab0-a6bd99ee46b8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:52.410132 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbda5eec-e7af-43c8-bab0-a6bd99ee46b8 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:52.517927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1231/2453] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:32:52 2022] POST /identity/v3/projects => generated 387 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:32:52.521371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8175661-8bc7-4268-b656-a50bc7606331 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:52.532709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8175661-8bc7-4268-b656-a50bc7606331 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:52.590081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7e3db0-7746-4787-b322-0bea7d3c0b67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:52.591291 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7e3db0-7746-4787-b322-0bea7d3c0b67 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:52.591425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7e3db0-7746-4787-b322-0bea7d3c0b67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:52.591705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7e3db0-7746-4787-b322-0bea7d3c0b67 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:52.592317 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7e3db0-7746-4787-b322-0bea7d3c0b67 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:52.594225 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7e3db0-7746-4787-b322-0bea7d3c0b67 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:52.693417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1223/2454] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:32:52 2022] POST /identity/v3/users => generated 294 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:32:52.697476 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbda5eec-e7af-43c8-bab0-a6bd99ee46b8 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:52.709122 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbda5eec-e7af-43c8-bab0-a6bd99ee46b8 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:52.714031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5fb448-b6e2-463c-b442-ae71fae18375 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:52.714653 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5fb448-b6e2-463c-b442-ae71fae18375 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:52.714930 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5fb448-b6e2-463c-b442-ae71fae18375 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:52.715201 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5fb448-b6e2-463c-b442-ae71fae18375 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:52.716387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5fb448-b6e2-463c-b442-ae71fae18375 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:52.717996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5fb448-b6e2-463c-b442-ae71fae18375 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:52.730075 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1232/2455] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:32:52 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:32:52.734972 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7e3db0-7746-4787-b322-0bea7d3c0b67 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:52.745391 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7e3db0-7746-4787-b322-0bea7d3c0b67 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:52.751337 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3cc325-3ef0-465c-bc6e-6f38e49e95c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:52.751941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3cc325-3ef0-465c-bc6e-6f38e49e95c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:52.752275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3cc325-3ef0-465c-bc6e-6f38e49e95c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:52.752550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3cc325-3ef0-465c-bc6e-6f38e49e95c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:52.753272 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3cc325-3ef0-465c-bc6e-6f38e49e95c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:52.754949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3cc325-3ef0-465c-bc6e-6f38e49e95c7 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:52.767343 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1224/2456] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:32:52 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:32:52.779222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5fb448-b6e2-463c-b442-ae71fae18375 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:52.784653 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:52.793071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5fb448-b6e2-463c-b442-ae71fae18375 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:52.799578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97ef36e-c454-434b-99ab-793214a2b97f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:52.800651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97ef36e-c454-434b-99ab-793214a2b97f admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:52.800651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97ef36e-c454-434b-99ab-793214a2b97f admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:52.800842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97ef36e-c454-434b-99ab-793214a2b97f admin admin] PATH_INFO: `/v3/projects/e0f1b8e948344836a36adcc55b964f14/users/89f89d2fb00e47e4bab6ffd136425ef2/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:52.817868 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97ef36e-c454-434b-99ab-793214a2b97f admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0f1b8e948344836a36adcc55b964f14, user_id=89f89d2fb00e47e4bab6ffd136425ef2, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:52.823013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97ef36e-c454-434b-99ab-793214a2b97f admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:52.947407 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1233/2457] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:32:52 2022] PUT /identity/v3/projects/e0f1b8e948344836a36adcc55b964f14/users/89f89d2fb00e47e4bab6ffd136425ef2/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:32:52.960155 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8669a4e7-c49a-45b4-a38c-1c8b0276eee1 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:52.960525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8669a4e7-c49a-45b4-a38c-1c8b0276eee1 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:52.960799 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8669a4e7-c49a-45b4-a38c-1c8b0276eee1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:52.995649 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-8669a4e7-c49a-45b4-a38c-1c8b0276eee1 None None] MFA Rules not processed for user `89f89d2fb00e47e4bab6ffd136425ef2`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:52.997087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8669a4e7-c49a-45b4-a38c-1c8b0276eee1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:53.066457 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1225/2458] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:52 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:53.071002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97ef36e-c454-434b-99ab-793214a2b97f admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:53.134448 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97ef36e-c454-434b-99ab-793214a2b97f admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:53.139943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecbe0be-da02-4f99-8c99-800412d51a0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:53.140606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecbe0be-da02-4f99-8c99-800412d51a0a admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:53.140883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecbe0be-da02-4f99-8c99-800412d51a0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:53.141151 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecbe0be-da02-4f99-8c99-800412d51a0a admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:53.141793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ecbe0be-da02-4f99-8c99-800412d51a0a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:53.143689 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ecbe0be-da02-4f99-8c99-800412d51a0a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:53.218640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1234/2459] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:32:53 2022] POST /identity/v3/projects => generated 387 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:32:53.228509 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8669a4e7-c49a-45b4-a38c-1c8b0276eee1 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:53.243288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8669a4e7-c49a-45b4-a38c-1c8b0276eee1 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:53.286886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:53.304777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399482d6-a681-415e-9297-845a133cfbdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:53.305066 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399482d6-a681-415e-9297-845a133cfbdf admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:53.305339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399482d6-a681-415e-9297-845a133cfbdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:53.305686 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399482d6-a681-415e-9297-845a133cfbdf admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:53.306332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399482d6-a681-415e-9297-845a133cfbdf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:53.308296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399482d6-a681-415e-9297-845a133cfbdf admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:53.382164 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1226/2460] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:32:53 2022] POST /identity/v3/users => generated 300 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:32:53.386382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecbe0be-da02-4f99-8c99-800412d51a0a admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:53.402796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecbe0be-da02-4f99-8c99-800412d51a0a admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:53.407920 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f241d363-2d1d-4cc9-9974-bf3ded6e7cc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:53.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f241d363-2d1d-4cc9-9974-bf3ded6e7cc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:53.408838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f241d363-2d1d-4cc9-9974-bf3ded6e7cc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:53.409102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f241d363-2d1d-4cc9-9974-bf3ded6e7cc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:53.409745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f241d363-2d1d-4cc9-9974-bf3ded6e7cc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:53.411368 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f241d363-2d1d-4cc9-9974-bf3ded6e7cc6 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:53.423473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1235/2461] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:32:53 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:32:53.427411 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399482d6-a681-415e-9297-845a133cfbdf admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:53.438072 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399482d6-a681-415e-9297-845a133cfbdf admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:53.442817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b92a93-7217-4df8-8636-0f9f18b32992 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:53.443478 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b92a93-7217-4df8-8636-0f9f18b32992 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:53.443754 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b92a93-7217-4df8-8636-0f9f18b32992 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:53.444027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b92a93-7217-4df8-8636-0f9f18b32992 admin admin] PATH_INFO: `/v3/projects/0b2fe4b528c64f51930dfa2336294d3d/users/ccb11f2e21bc49c3b444868a8b6a08d2/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:53.461477 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b92a93-7217-4df8-8636-0f9f18b32992 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b2fe4b528c64f51930dfa2336294d3d, user_id=ccb11f2e21bc49c3b444868a8b6a08d2, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:53.466243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b92a93-7217-4df8-8636-0f9f18b32992 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:53.554823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1227/2462] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:32:53 2022] PUT /identity/v3/projects/0b2fe4b528c64f51930dfa2336294d3d/users/ccb11f2e21bc49c3b444868a8b6a08d2/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:53.556787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f241d363-2d1d-4cc9-9974-bf3ded6e7cc6 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:53.574910 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f241d363-2d1d-4cc9-9974-bf3ded6e7cc6 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:53.647236 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f34cbc7-5bae-4bbf-8082-bf5c05c10e22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:53.648639 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f34cbc7-5bae-4bbf-8082-bf5c05c10e22 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:53.648918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f34cbc7-5bae-4bbf-8082-bf5c05c10e22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:53.649190 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f34cbc7-5bae-4bbf-8082-bf5c05c10e22 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:53.649843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f34cbc7-5bae-4bbf-8082-bf5c05c10e22 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:53.651503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f34cbc7-5bae-4bbf-8082-bf5c05c10e22 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:53.662924 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1236/2463] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:32:53 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:32:53.670164 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b92a93-7217-4df8-8636-0f9f18b32992 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:53.681318 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b92a93-7217-4df8-8636-0f9f18b32992 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:53.690015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca12fd4-c385-4870-aa9e-ddeb5cabb708 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:53.690792 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca12fd4-c385-4870-aa9e-ddeb5cabb708 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:53.691435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca12fd4-c385-4870-aa9e-ddeb5cabb708 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:53.691716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca12fd4-c385-4870-aa9e-ddeb5cabb708 admin admin] PATH_INFO: `/v3/projects/0b2fe4b528c64f51930dfa2336294d3d/users/ccb11f2e21bc49c3b444868a8b6a08d2/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:53.694154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca12fd4-c385-4870-aa9e-ddeb5cabb708 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b2fe4b528c64f51930dfa2336294d3d, user_id=ccb11f2e21bc49c3b444868a8b6a08d2, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:53.696924 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca12fd4-c385-4870-aa9e-ddeb5cabb708 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:53.708982 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1228/2464] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:32:53 2022] PUT /identity/v3/projects/0b2fe4b528c64f51930dfa2336294d3d/users/ccb11f2e21bc49c3b444868a8b6a08d2/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:32:53.723803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95ce3af-d072-4559-a7f3-9f9f80cc922e None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:53.724095 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95ce3af-d072-4559-a7f3-9f9f80cc922e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:53.724413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95ce3af-d072-4559-a7f3-9f9f80cc922e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:53.755040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-a95ce3af-d072-4559-a7f3-9f9f80cc922e None None] MFA Rules not processed for user `ccb11f2e21bc49c3b444868a8b6a08d2`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:32:53.756322 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-a95ce3af-d072-4559-a7f3-9f9f80cc922e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:53.805282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:53.865691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1237/2465] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:32:53 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:32:53.880470 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca12fd4-c385-4870-aa9e-ddeb5cabb708 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:53.887681 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7ca12fd4-c385-4870-aa9e-ddeb5cabb708 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:53.967882 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca12fd4-c385-4870-aa9e-ddeb5cabb708 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:53.972064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32c6d7a-a62d-439c-84f4-8ca813aa5e6d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:53.972731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32c6d7a-a62d-439c-84f4-8ca813aa5e6d service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:53.973005 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32c6d7a-a62d-439c-84f4-8ca813aa5e6d service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:53.973269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32c6d7a-a62d-439c-84f4-8ca813aa5e6d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:53.984600 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32c6d7a-a62d-439c-84f4-8ca813aa5e6d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:53.986059 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32c6d7a-a62d-439c-84f4-8ca813aa5e6d service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:53.998072 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1229/2466] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:32:53 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:32:55.417121 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95ce3af-d072-4559-a7f3-9f9f80cc922e None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:55.419227 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-a95ce3af-d072-4559-a7f3-9f9f80cc922e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:55.437128 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95ce3af-d072-4559-a7f3-9f9f80cc922e None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:55.444404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a986cad3-3455-4600-bb85-27475a0b1d09 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:55.445303 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a986cad3-3455-4600-bb85-27475a0b1d09 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:55.445873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a986cad3-3455-4600-bb85-27475a0b1d09 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:55.446306 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a986cad3-3455-4600-bb85-27475a0b1d09 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:55.534976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a986cad3-3455-4600-bb85-27475a0b1d09 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:55.536425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a986cad3-3455-4600-bb85-27475a0b1d09 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:55.547466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1238/2467] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:32:55 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:32:59.016499 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32c6d7a-a62d-439c-84f4-8ca813aa5e6d service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:32:59.017903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e32c6d7a-a62d-439c-84f4-8ca813aa5e6d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:59.079923 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32c6d7a-a62d-439c-84f4-8ca813aa5e6d service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:32:59.084029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c79fcee-9c54-433d-8d5e-f0dda965c101 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:32:59.084666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c79fcee-9c54-433d-8d5e-f0dda965c101 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:32:59.084938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c79fcee-9c54-433d-8d5e-f0dda965c101 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:32:59.085202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c79fcee-9c54-433d-8d5e-f0dda965c101 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:32:59.087072 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8c79fcee-9c54-433d-8d5e-f0dda965c101 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:32:59.143897 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:32:59.159576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c79fcee-9c54-433d-8d5e-f0dda965c101 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:32:59.161019 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c79fcee-9c54-433d-8d5e-f0dda965c101 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:32:59.175911 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1230/2468] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:32:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:00.513065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a986cad3-3455-4600-bb85-27475a0b1d09 service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:00.523157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a986cad3-3455-4600-bb85-27475a0b1d09 service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:00.527335 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d8831d-f57e-46b4-aba7-e07a87ece131 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:00.528204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d8831d-f57e-46b4-aba7-e07a87ece131 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:00.528570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d8831d-f57e-46b4-aba7-e07a87ece131 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:00.528800 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d8831d-f57e-46b4-aba7-e07a87ece131 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:00.596172 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d8831d-f57e-46b4-aba7-e07a87ece131 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:00.597661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d8831d-f57e-46b4-aba7-e07a87ece131 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:00.610176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1239/2469] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:00 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:02.831281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c79fcee-9c54-433d-8d5e-f0dda965c101 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:02.832735 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8c79fcee-9c54-433d-8d5e-f0dda965c101 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:02.905508 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c79fcee-9c54-433d-8d5e-f0dda965c101 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:02.911067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03ee620-4241-49c8-bf8b-24240b3da02a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '972b0f1dbfef4d85a6dfc4c5fd839bde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:02.911718 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03ee620-4241-49c8-bf8b-24240b3da02a service placement] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:02.911977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03ee620-4241-49c8-bf8b-24240b3da02a service placement] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:02.912293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03ee620-4241-49c8-bf8b-24240b3da02a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:02.914077 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-c03ee620-4241-49c8-bf8b-24240b3da02a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:02.926741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03ee620-4241-49c8-bf8b-24240b3da02a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:02.928366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03ee620-4241-49c8-bf8b-24240b3da02a service placement] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:02.939244 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1231/2470] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:02 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:02.952244 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d8831d-f57e-46b4-aba7-e07a87ece131 service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:02.954577 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-65d8831d-f57e-46b4-aba7-e07a87ece131 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:02.973825 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d8831d-f57e-46b4-aba7-e07a87ece131 service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:02.979844 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ebea99-a435-47b2-ab05-9deef1560c06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:02.980586 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ebea99-a435-47b2-ab05-9deef1560c06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:02.981076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ebea99-a435-47b2-ab05-9deef1560c06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:02.981693 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ebea99-a435-47b2-ab05-9deef1560c06 admin admin] PATH_INFO: `/v3/users/0f4a35c46e4741c1b238f24d203b416d` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:02.982996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ebea99-a435-47b2-ab05-9deef1560c06 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0f4a35c46e4741c1b238f24d203b416d)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:02.985051 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ebea99-a435-47b2-ab05-9deef1560c06 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:03.228046 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-c6ebea99-a435-47b2-ab05-9deef1560c06 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0f4a35c46e4741c1b238f24d203b416d'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:03.235480 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-c6ebea99-a435-47b2-ab05-9deef1560c06 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0f4a35c46e4741c1b238f24d203b416d'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:03.243066 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:03.249186 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-c6ebea99-a435-47b2-ab05-9deef1560c06 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0f4a35c46e4741c1b238f24d203b416d'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:03.340185 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-c6ebea99-a435-47b2-ab05-9deef1560c06 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0f4a35c46e4741c1b238f24d203b416d'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:03.340846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-c6ebea99-a435-47b2-ab05-9deef1560c06 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0f4a35c46e4741c1b238f24d203b416d'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:03.351894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1240/2471] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:02 2022] DELETE /identity/v3/users/0f4a35c46e4741c1b238f24d203b416d => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:03.622117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03ee620-4241-49c8-bf8b-24240b3da02a service placement] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:03.640365 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03ee620-4241-49c8-bf8b-24240b3da02a service placement] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:03.717412 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd39da7-4408-4608-b9c1-5242be949eab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:03.718112 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd39da7-4408-4608-b9c1-5242be949eab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:03.718389 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd39da7-4408-4608-b9c1-5242be949eab admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:03.718658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd39da7-4408-4608-b9c1-5242be949eab admin admin] PATH_INFO: `/v3/projects/13b534c485db48d8a37e024702f87092` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:03.719825 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd39da7-4408-4608-b9c1-5242be949eab admin admin] RBAC: Authorizing `identity:delete_project(project_id=13b534c485db48d8a37e024702f87092)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:03.721745 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd39da7-4408-4608-b9c1-5242be949eab admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:03.841704 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0cd39da7-4408-4608-b9c1-5242be949eab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '13b534c485db48d8a37e024702f87092', 'request_id': 'req-0cd39da7-4408-4608-b9c1-5242be949eab'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:03.848006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0cd39da7-4408-4608-b9c1-5242be949eab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '13b534c485db48d8a37e024702f87092', 'request_id': 'req-0cd39da7-4408-4608-b9c1-5242be949eab'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:03.854615 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0cd39da7-4408-4608-b9c1-5242be949eab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '13b534c485db48d8a37e024702f87092', 'request_id': 'req-0cd39da7-4408-4608-b9c1-5242be949eab'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:03.933877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0cd39da7-4408-4608-b9c1-5242be949eab admin admin] The token cache is being invalidate because project 13b534c485db48d8a37e024702f87092 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:03.934160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0cd39da7-4408-4608-b9c1-5242be949eab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:03.934922 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0cd39da7-4408-4608-b9c1-5242be949eab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:03.936504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1232/2472] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:03 2022] DELETE /identity/v3/projects/13b534c485db48d8a37e024702f87092 => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:03.941775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee563ad-3901-4969-9901-79501ce02347 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:03.942201 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee563ad-3901-4969-9901-79501ce02347 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:03.942675 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee563ad-3901-4969-9901-79501ce02347 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:03.963620 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-1ee563ad-3901-4969-9901-79501ce02347 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:03.964889 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1ee563ad-3901-4969-9901-79501ce02347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:04.030744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1241/2473] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:04.039611 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e50b96-13df-47e3-970b-007324c54312 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:04.039951 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e50b96-13df-47e3-970b-007324c54312 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:04.040252 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e50b96-13df-47e3-970b-007324c54312 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:04.056648 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:04.062275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-34e50b96-13df-47e3-970b-007324c54312 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:04.063497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-34e50b96-13df-47e3-970b-007324c54312 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:04.071730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1233/2474] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:04.076082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee563ad-3901-4969-9901-79501ce02347 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:04.086365 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee563ad-3901-4969-9901-79501ce02347 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:04.091525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1380058-e772-408a-ab44-6b99a9d7ad3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:04.092968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1380058-e772-408a-ab44-6b99a9d7ad3e admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:04.093306 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1380058-e772-408a-ab44-6b99a9d7ad3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:04.093801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1380058-e772-408a-ab44-6b99a9d7ad3e admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:04.094565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1380058-e772-408a-ab44-6b99a9d7ad3e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:33:04.094959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1380058-e772-408a-ab44-6b99a9d7ad3e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:04.101462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1380058-e772-408a-ab44-6b99a9d7ad3e admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:04.113513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1242/2475] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:33:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:33:04.118060 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e50b96-13df-47e3-970b-007324c54312 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:04.127499 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e50b96-13df-47e3-970b-007324c54312 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:04.133203 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72507c85-3f61-4673-8981-b5beadf1e0b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:04.133817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72507c85-3f61-4673-8981-b5beadf1e0b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:04.140761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72507c85-3f61-4673-8981-b5beadf1e0b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:04.140761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72507c85-3f61-4673-8981-b5beadf1e0b9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:04.140761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72507c85-3f61-4673-8981-b5beadf1e0b9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:04.140761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72507c85-3f61-4673-8981-b5beadf1e0b9 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:04.228422 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1234/2476] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:33:04 2022] POST /identity/v3/projects => generated 365 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:04.232538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1380058-e772-408a-ab44-6b99a9d7ad3e admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:04.242766 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1380058-e772-408a-ab44-6b99a9d7ad3e admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:04.300700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b187a76d-4725-4159-9a82-3b9f9d4e74b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:04.301220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b187a76d-4725-4159-9a82-3b9f9d4e74b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:04.301494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b187a76d-4725-4159-9a82-3b9f9d4e74b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:04.301781 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b187a76d-4725-4159-9a82-3b9f9d4e74b8 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:04.302505 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b187a76d-4725-4159-9a82-3b9f9d4e74b8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:04.304533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b187a76d-4725-4159-9a82-3b9f9d4e74b8 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:04.414439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1243/2477] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:33:04 2022] POST /identity/v3/users => generated 283 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:04.418963 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72507c85-3f61-4673-8981-b5beadf1e0b9 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:04.429484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72507c85-3f61-4673-8981-b5beadf1e0b9 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:04.434380 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fd1e8c-860f-45da-b033-6f6bfe877ce0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:04.434986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fd1e8c-860f-45da-b033-6f6bfe877ce0 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:04.435264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fd1e8c-860f-45da-b033-6f6bfe877ce0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:04.435560 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fd1e8c-860f-45da-b033-6f6bfe877ce0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:04.436396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6fd1e8c-860f-45da-b033-6f6bfe877ce0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:04.438016 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6fd1e8c-860f-45da-b033-6f6bfe877ce0 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:04.448440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1235/2478] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:04.453117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b187a76d-4725-4159-9a82-3b9f9d4e74b8 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:04.463426 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b187a76d-4725-4159-9a82-3b9f9d4e74b8 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:04.468741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26ab1d8-41f5-490f-bbd0-53eb54689836 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:04.469324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26ab1d8-41f5-490f-bbd0-53eb54689836 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:04.469601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26ab1d8-41f5-490f-bbd0-53eb54689836 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:04.470713 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26ab1d8-41f5-490f-bbd0-53eb54689836 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:04.470713 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26ab1d8-41f5-490f-bbd0-53eb54689836 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:04.472088 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26ab1d8-41f5-490f-bbd0-53eb54689836 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:04.482579 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1244/2479] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:04 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:04.486697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fd1e8c-860f-45da-b033-6f6bfe877ce0 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:04.496799 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fd1e8c-860f-45da-b033-6f6bfe877ce0 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:04.502693 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb959c9-6cd3-4982-a708-e91764773861 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:04.503342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb959c9-6cd3-4982-a708-e91764773861 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:04.503617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb959c9-6cd3-4982-a708-e91764773861 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:04.503884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb959c9-6cd3-4982-a708-e91764773861 admin admin] PATH_INFO: `/v3/projects/bde084df400f4e93b631a21a0acddf08/users/986d41bb84464d4588f719063fe25ff9/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:04.519315 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb959c9-6cd3-4982-a708-e91764773861 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bde084df400f4e93b631a21a0acddf08, user_id=986d41bb84464d4588f719063fe25ff9, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:04.522030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb959c9-6cd3-4982-a708-e91764773861 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:04.586090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1236/2480] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:04 2022] PUT /identity/v3/projects/bde084df400f4e93b631a21a0acddf08/users/986d41bb84464d4588f719063fe25ff9/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:04.596826 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca2817c-bb8b-438f-b605-f42ee568840a None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:04.597102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca2817c-bb8b-438f-b605-f42ee568840a None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:04.597375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca2817c-bb8b-438f-b605-f42ee568840a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:04.625252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-bca2817c-bb8b-438f-b605-f42ee568840a None None] MFA Rules not processed for user `986d41bb84464d4588f719063fe25ff9`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:04.626445 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-bca2817c-bb8b-438f-b605-f42ee568840a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:04.664291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:04.691324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1245/2481] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:04 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:04.703368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb959c9-6cd3-4982-a708-e91764773861 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:04.705091 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7fb959c9-6cd3-4982-a708-e91764773861 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:04.764447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb959c9-6cd3-4982-a708-e91764773861 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:04.768670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f45ceab-2e7c-447f-976c-9e31c0c669a7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:04.772627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f45ceab-2e7c-447f-976c-9e31c0c669a7 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:04.772627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f45ceab-2e7c-447f-976c-9e31c0c669a7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:04.772627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f45ceab-2e7c-447f-976c-9e31c0c669a7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:04.778206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f45ceab-2e7c-447f-976c-9e31c0c669a7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:04.779693 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f45ceab-2e7c-447f-976c-9e31c0c669a7 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:04.788748 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1237/2482] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:04 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:04.902066 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca2817c-bb8b-438f-b605-f42ee568840a None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:04.903595 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-bca2817c-bb8b-438f-b605-f42ee568840a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:04.972587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca2817c-bb8b-438f-b605-f42ee568840a None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:04.977296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd804be-d323-47b4-af0e-c8b06e17face admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:04.977886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd804be-d323-47b4-af0e-c8b06e17face admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:04.978164 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd804be-d323-47b4-af0e-c8b06e17face admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:04.978635 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd804be-d323-47b4-af0e-c8b06e17face admin admin] PATH_INFO: `/v3/users/228e7e06d92341cab263dcbf6bc6647e` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:04.979777 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd804be-d323-47b4-af0e-c8b06e17face admin admin] RBAC: Authorizing `identity:delete_user(user_id=228e7e06d92341cab263dcbf6bc6647e)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:04.981739 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd804be-d323-47b4-af0e-c8b06e17face admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:05.194054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-1cd804be-d323-47b4-af0e-c8b06e17face admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '228e7e06d92341cab263dcbf6bc6647e'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:05.201121 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-1cd804be-d323-47b4-af0e-c8b06e17face admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '228e7e06d92341cab263dcbf6bc6647e'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:05.213961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-1cd804be-d323-47b4-af0e-c8b06e17face admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '228e7e06d92341cab263dcbf6bc6647e'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:05.288783 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-1cd804be-d323-47b4-af0e-c8b06e17face admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '228e7e06d92341cab263dcbf6bc6647e'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:05.289437 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-1cd804be-d323-47b4-af0e-c8b06e17face admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '228e7e06d92341cab263dcbf6bc6647e'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:05.295955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1246/2483] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:04 2022] DELETE /identity/v3/users/228e7e06d92341cab263dcbf6bc6647e => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:05.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2cdedf-be47-48f5-879b-d3b9a08468b4 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:05.301462 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2cdedf-be47-48f5-879b-d3b9a08468b4 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:05.301585 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2cdedf-be47-48f5-879b-d3b9a08468b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:05.321205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-eb2cdedf-be47-48f5-879b-d3b9a08468b4 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:05.322446 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-eb2cdedf-be47-48f5-879b-d3b9a08468b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:05.382169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1238/2484] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:05.392308 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd804be-d323-47b4-af0e-c8b06e17face admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:05.393712 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1cd804be-d323-47b4-af0e-c8b06e17face admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:05.452061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd804be-d323-47b4-af0e-c8b06e17face admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:05.456058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8ba4c9-d7bb-45d2-9b8d-d4056ba00a29 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:05.456695 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8ba4c9-d7bb-45d2-9b8d-d4056ba00a29 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:05.456971 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8ba4c9-d7bb-45d2-9b8d-d4056ba00a29 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:05.457324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8ba4c9-d7bb-45d2-9b8d-d4056ba00a29 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:05.466321 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8ba4c9-d7bb-45d2-9b8d-d4056ba00a29 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:05.469608 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8ba4c9-d7bb-45d2-9b8d-d4056ba00a29 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:05.474076 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:05.491764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1247/2485] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:05.987471 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2cdedf-be47-48f5-879b-d3b9a08468b4 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:05.988926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-eb2cdedf-be47-48f5-879b-d3b9a08468b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:06.002307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2cdedf-be47-48f5-879b-d3b9a08468b4 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:06.007066 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:06.008198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:06.008198 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:06.008409 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb admin admin] PATH_INFO: `/v3/projects/3a0243a7f1ff44a1bb6fdfda13433961` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:06.009483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a0243a7f1ff44a1bb6fdfda13433961)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:06.011963 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:06.095187 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:06.101060 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a0243a7f1ff44a1bb6fdfda13433961', 'request_id': 'req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:06.106588 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a0243a7f1ff44a1bb6fdfda13433961', 'request_id': 'req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:06.112884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a0243a7f1ff44a1bb6fdfda13433961', 'request_id': 'req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:06.194020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb admin admin] The token cache is being invalidate because project 3a0243a7f1ff44a1bb6fdfda13433961 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:06.194515 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:06.195458 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:06.197732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1239/2486] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:05 2022] DELETE /identity/v3/projects/3a0243a7f1ff44a1bb6fdfda13433961 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:06.202099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8ba4c9-d7bb-45d2-9b8d-d4056ba00a29 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:06.212922 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8ba4c9-d7bb-45d2-9b8d-d4056ba00a29 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:06.268836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968f8ad2-2add-4f3e-a30b-bf8cbe667878 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:06.269383 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968f8ad2-2add-4f3e-a30b-bf8cbe667878 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:06.269655 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968f8ad2-2add-4f3e-a30b-bf8cbe667878 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:06.270008 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968f8ad2-2add-4f3e-a30b-bf8cbe667878 admin admin] PATH_INFO: `/v3/users/89f89d2fb00e47e4bab6ffd136425ef2` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:06.271116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968f8ad2-2add-4f3e-a30b-bf8cbe667878 admin admin] RBAC: Authorizing `identity:delete_user(user_id=89f89d2fb00e47e4bab6ffd136425ef2)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:06.273169 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968f8ad2-2add-4f3e-a30b-bf8cbe667878 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:06.427017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-968f8ad2-2add-4f3e-a30b-bf8cbe667878 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89f89d2fb00e47e4bab6ffd136425ef2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:06.433890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-968f8ad2-2add-4f3e-a30b-bf8cbe667878 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89f89d2fb00e47e4bab6ffd136425ef2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:06.446095 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-968f8ad2-2add-4f3e-a30b-bf8cbe667878 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89f89d2fb00e47e4bab6ffd136425ef2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:06.528539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-968f8ad2-2add-4f3e-a30b-bf8cbe667878 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89f89d2fb00e47e4bab6ffd136425ef2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:06.529140 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-968f8ad2-2add-4f3e-a30b-bf8cbe667878 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89f89d2fb00e47e4bab6ffd136425ef2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:06.535500 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1248/2487] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:06 2022] DELETE /identity/v3/users/89f89d2fb00e47e4bab6ffd136425ef2 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:07.154439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:07.155930 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:07.169404 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d84435-8cda-4945-a6dd-cbbe3aacd6cb admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:07.229608 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22526b2-91f9-482d-b98c-53cb0ecea82b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:07.230286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22526b2-91f9-482d-b98c-53cb0ecea82b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:07.230561 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22526b2-91f9-482d-b98c-53cb0ecea82b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:07.230827 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22526b2-91f9-482d-b98c-53cb0ecea82b admin admin] PATH_INFO: `/v3/projects/e0f1b8e948344836a36adcc55b964f14` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:07.231946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22526b2-91f9-482d-b98c-53cb0ecea82b admin admin] RBAC: Authorizing `identity:delete_project(project_id=e0f1b8e948344836a36adcc55b964f14)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:07.233870 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22526b2-91f9-482d-b98c-53cb0ecea82b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:07.350691 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f22526b2-91f9-482d-b98c-53cb0ecea82b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e0f1b8e948344836a36adcc55b964f14', 'request_id': 'req-f22526b2-91f9-482d-b98c-53cb0ecea82b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:07.360374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f22526b2-91f9-482d-b98c-53cb0ecea82b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e0f1b8e948344836a36adcc55b964f14', 'request_id': 'req-f22526b2-91f9-482d-b98c-53cb0ecea82b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:07.366796 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f22526b2-91f9-482d-b98c-53cb0ecea82b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e0f1b8e948344836a36adcc55b964f14', 'request_id': 'req-f22526b2-91f9-482d-b98c-53cb0ecea82b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:07.448680 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f22526b2-91f9-482d-b98c-53cb0ecea82b admin admin] The token cache is being invalidate because project e0f1b8e948344836a36adcc55b964f14 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:07.448993 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f22526b2-91f9-482d-b98c-53cb0ecea82b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:07.449614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f22526b2-91f9-482d-b98c-53cb0ecea82b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:07.451538 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1240/2488] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:07 2022] DELETE /identity/v3/projects/e0f1b8e948344836a36adcc55b964f14 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:07.455786 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968f8ad2-2add-4f3e-a30b-bf8cbe667878 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:07.457292 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-968f8ad2-2add-4f3e-a30b-bf8cbe667878 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:07.471568 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968f8ad2-2add-4f3e-a30b-bf8cbe667878 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:07.529592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c68f39b-bc13-44a7-882b-24ce73697a29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:07.530215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c68f39b-bc13-44a7-882b-24ce73697a29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:07.530488 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c68f39b-bc13-44a7-882b-24ce73697a29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:07.530756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c68f39b-bc13-44a7-882b-24ce73697a29 admin admin] PATH_INFO: `/v3/users/ccb11f2e21bc49c3b444868a8b6a08d2` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:07.531859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c68f39b-bc13-44a7-882b-24ce73697a29 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ccb11f2e21bc49c3b444868a8b6a08d2)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:07.534446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c68f39b-bc13-44a7-882b-24ce73697a29 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:07.555475 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:07.735896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9c68f39b-bc13-44a7-882b-24ce73697a29 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ccb11f2e21bc49c3b444868a8b6a08d2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:07.742885 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9c68f39b-bc13-44a7-882b-24ce73697a29 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ccb11f2e21bc49c3b444868a8b6a08d2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:07.754511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9c68f39b-bc13-44a7-882b-24ce73697a29 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ccb11f2e21bc49c3b444868a8b6a08d2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:07.877677 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9c68f39b-bc13-44a7-882b-24ce73697a29 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ccb11f2e21bc49c3b444868a8b6a08d2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:07.878242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9c68f39b-bc13-44a7-882b-24ce73697a29 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ccb11f2e21bc49c3b444868a8b6a08d2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:07.884583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1249/2489] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:07 2022] DELETE /identity/v3/users/ccb11f2e21bc49c3b444868a8b6a08d2 => generated 0 bytes in 430 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:08.421376 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22526b2-91f9-482d-b98c-53cb0ecea82b admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:08.423409 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f22526b2-91f9-482d-b98c-53cb0ecea82b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:08.449290 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22526b2-91f9-482d-b98c-53cb0ecea82b admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:08.508862 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dedbb29-0fcf-4338-adc2-a663c0e8b966 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:08.509691 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dedbb29-0fcf-4338-adc2-a663c0e8b966 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:08.510039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dedbb29-0fcf-4338-adc2-a663c0e8b966 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:08.510418 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dedbb29-0fcf-4338-adc2-a663c0e8b966 admin admin] PATH_INFO: `/v3/projects/0b2fe4b528c64f51930dfa2336294d3d` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:08.511497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dedbb29-0fcf-4338-adc2-a663c0e8b966 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b2fe4b528c64f51930dfa2336294d3d)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:08.513617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dedbb29-0fcf-4338-adc2-a663c0e8b966 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:08.515145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:08.622756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5dedbb29-0fcf-4338-adc2-a663c0e8b966 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b2fe4b528c64f51930dfa2336294d3d', 'request_id': 'req-5dedbb29-0fcf-4338-adc2-a663c0e8b966'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:08.629419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5dedbb29-0fcf-4338-adc2-a663c0e8b966 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b2fe4b528c64f51930dfa2336294d3d', 'request_id': 'req-5dedbb29-0fcf-4338-adc2-a663c0e8b966'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:08.638800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5dedbb29-0fcf-4338-adc2-a663c0e8b966 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b2fe4b528c64f51930dfa2336294d3d', 'request_id': 'req-5dedbb29-0fcf-4338-adc2-a663c0e8b966'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:08.740991 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5dedbb29-0fcf-4338-adc2-a663c0e8b966 admin admin] The token cache is being invalidate because project 0b2fe4b528c64f51930dfa2336294d3d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:08.741282 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5dedbb29-0fcf-4338-adc2-a663c0e8b966 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:08.741910 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-5dedbb29-0fcf-4338-adc2-a663c0e8b966 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:08.743462 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1241/2490] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:08 2022] DELETE /identity/v3/projects/0b2fe4b528c64f51930dfa2336294d3d => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:08.749431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640088a1-728e-484b-86be-340f1fe2ffea None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:08.750458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640088a1-728e-484b-86be-340f1fe2ffea None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:08.750458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640088a1-728e-484b-86be-340f1fe2ffea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:08.769668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-640088a1-728e-484b-86be-340f1fe2ffea None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:08.770871 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-640088a1-728e-484b-86be-340f1fe2ffea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:08.833503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1250/2491] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:08.842015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610b0946-c8ca-4d5a-a2fa-d4ac3b3aa2e2 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:08.842298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610b0946-c8ca-4d5a-a2fa-d4ac3b3aa2e2 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:08.842568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610b0946-c8ca-4d5a-a2fa-d4ac3b3aa2e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:08.863527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-610b0946-c8ca-4d5a-a2fa-d4ac3b3aa2e2 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:08.865206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-610b0946-c8ca-4d5a-a2fa-d4ac3b3aa2e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:08.872906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1242/2492] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:08.877437 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640088a1-728e-484b-86be-340f1fe2ffea None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:08.887778 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640088a1-728e-484b-86be-340f1fe2ffea None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:08.893881 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d77dbb-f3c0-4b83-ac55-fb4fa24ee059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:08.894497 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d77dbb-f3c0-4b83-ac55-fb4fa24ee059 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:08.894780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d77dbb-f3c0-4b83-ac55-fb4fa24ee059 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:08.895118 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d77dbb-f3c0-4b83-ac55-fb4fa24ee059 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:08.895767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d77dbb-f3c0-4b83-ac55-fb4fa24ee059 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:33:08.896066 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d77dbb-f3c0-4b83-ac55-fb4fa24ee059 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:08.897737 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d77dbb-f3c0-4b83-ac55-fb4fa24ee059 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:08.910398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1251/2493] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:33:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:33:08.914974 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610b0946-c8ca-4d5a-a2fa-d4ac3b3aa2e2 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:08.925181 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610b0946-c8ca-4d5a-a2fa-d4ac3b3aa2e2 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:08.929829 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38fed7a-46d7-43be-b86d-ab2983f41241 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:08.930423 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38fed7a-46d7-43be-b86d-ab2983f41241 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:08.930694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38fed7a-46d7-43be-b86d-ab2983f41241 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:08.930962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38fed7a-46d7-43be-b86d-ab2983f41241 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:08.931574 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38fed7a-46d7-43be-b86d-ab2983f41241 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:08.933512 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38fed7a-46d7-43be-b86d-ab2983f41241 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:09.028594 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1243/2494] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:33:08 2022] POST /identity/v3/projects => generated 393 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:09.033493 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d77dbb-f3c0-4b83-ac55-fb4fa24ee059 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:09.043891 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d77dbb-f3c0-4b83-ac55-fb4fa24ee059 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:09.100935 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be57c023-964f-41b6-b1e1-c1be9cd73d6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:09.101536 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be57c023-964f-41b6-b1e1-c1be9cd73d6e admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:09.101814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be57c023-964f-41b6-b1e1-c1be9cd73d6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:09.102089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be57c023-964f-41b6-b1e1-c1be9cd73d6e admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:09.102721 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be57c023-964f-41b6-b1e1-c1be9cd73d6e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:09.104709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be57c023-964f-41b6-b1e1-c1be9cd73d6e admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:09.189689 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1252/2495] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:33:09 2022] POST /identity/v3/users => generated 297 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:09.195112 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38fed7a-46d7-43be-b86d-ab2983f41241 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:09.208004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38fed7a-46d7-43be-b86d-ab2983f41241 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:09.212958 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247b3095-55e4-4622-bc44-2cb2a211794e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:09.221896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247b3095-55e4-4622-bc44-2cb2a211794e admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:09.221896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247b3095-55e4-4622-bc44-2cb2a211794e admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:09.221896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247b3095-55e4-4622-bc44-2cb2a211794e admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:09.221896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247b3095-55e4-4622-bc44-2cb2a211794e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:09.221896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247b3095-55e4-4622-bc44-2cb2a211794e admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:09.229098 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1244/2496] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:09 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:09.234022 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be57c023-964f-41b6-b1e1-c1be9cd73d6e admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:09.244906 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be57c023-964f-41b6-b1e1-c1be9cd73d6e admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:09.250561 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ab54e4-1b22-40f9-86cb-b16d343ec14e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:09.250561 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ab54e4-1b22-40f9-86cb-b16d343ec14e admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:09.250561 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ab54e4-1b22-40f9-86cb-b16d343ec14e admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:09.250950 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ab54e4-1b22-40f9-86cb-b16d343ec14e admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:09.251421 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ab54e4-1b22-40f9-86cb-b16d343ec14e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:09.253030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ab54e4-1b22-40f9-86cb-b16d343ec14e admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:09.264545 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1253/2497] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:09.270117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247b3095-55e4-4622-bc44-2cb2a211794e admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:09.280880 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247b3095-55e4-4622-bc44-2cb2a211794e admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:09.287259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e380a2f1-5b21-4de9-a88f-ccc85c996508 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:09.287904 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e380a2f1-5b21-4de9-a88f-ccc85c996508 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:09.288224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e380a2f1-5b21-4de9-a88f-ccc85c996508 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:09.288495 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e380a2f1-5b21-4de9-a88f-ccc85c996508 admin admin] PATH_INFO: `/v3/projects/995b9d4778db4364b42baa7b843025d8/users/77368260eb8b463681ccea01dcf84afd/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:09.306315 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e380a2f1-5b21-4de9-a88f-ccc85c996508 admin admin] RBAC: Authorizing `identity:create_grant(project_id=995b9d4778db4364b42baa7b843025d8, user_id=77368260eb8b463681ccea01dcf84afd, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:09.309046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e380a2f1-5b21-4de9-a88f-ccc85c996508 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:09.413291 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1245/2498] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:09 2022] PUT /identity/v3/projects/995b9d4778db4364b42baa7b843025d8/users/77368260eb8b463681ccea01dcf84afd/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:09.424985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd2bf3b-7fca-4a78-aab6-a39a50625b07 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:09.425322 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd2bf3b-7fca-4a78-aab6-a39a50625b07 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:09.425600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd2bf3b-7fca-4a78-aab6-a39a50625b07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:09.434221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:09.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-4cd2bf3b-7fca-4a78-aab6-a39a50625b07 None None] MFA Rules not processed for user `77368260eb8b463681ccea01dcf84afd`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:09.457239 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-4cd2bf3b-7fca-4a78-aab6-a39a50625b07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:09.522622 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1254/2499] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:09 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:09.532601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e380a2f1-5b21-4de9-a88f-ccc85c996508 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:09.593876 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e380a2f1-5b21-4de9-a88f-ccc85c996508 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:09.598496 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186c4341-c5e7-40a3-8b89-dfb4525cffe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:09.599123 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186c4341-c5e7-40a3-8b89-dfb4525cffe5 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:09.599397 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186c4341-c5e7-40a3-8b89-dfb4525cffe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:09.599666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186c4341-c5e7-40a3-8b89-dfb4525cffe5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:09.600315 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186c4341-c5e7-40a3-8b89-dfb4525cffe5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:09.602206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186c4341-c5e7-40a3-8b89-dfb4525cffe5 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:09.700341 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1246/2500] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:33:09 2022] POST /identity/v3/projects => generated 391 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:09.705387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd2bf3b-7fca-4a78-aab6-a39a50625b07 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:09.716542 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd2bf3b-7fca-4a78-aab6-a39a50625b07 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:09.773463 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edcf792-7576-4654-a75b-9a72fc52f6bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:09.774111 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edcf792-7576-4654-a75b-9a72fc52f6bc admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:09.774302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edcf792-7576-4654-a75b-9a72fc52f6bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:09.774573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edcf792-7576-4654-a75b-9a72fc52f6bc admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:09.775202 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edcf792-7576-4654-a75b-9a72fc52f6bc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:09.777283 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edcf792-7576-4654-a75b-9a72fc52f6bc admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:09.900662 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1255/2501] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:33:09 2022] POST /identity/v3/users => generated 302 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:33:09.904752 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186c4341-c5e7-40a3-8b89-dfb4525cffe5 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:09.915306 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186c4341-c5e7-40a3-8b89-dfb4525cffe5 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:09.919931 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1a279e-5090-49ab-b117-72413ba05371 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:09.920566 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1a279e-5090-49ab-b117-72413ba05371 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:09.920843 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1a279e-5090-49ab-b117-72413ba05371 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:09.921115 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1a279e-5090-49ab-b117-72413ba05371 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:09.921747 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1a279e-5090-49ab-b117-72413ba05371 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:09.923840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1a279e-5090-49ab-b117-72413ba05371 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:09.929385 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:09.935052 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1247/2502] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:09.939864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edcf792-7576-4654-a75b-9a72fc52f6bc admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:09.949813 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edcf792-7576-4654-a75b-9a72fc52f6bc admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:09.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd60737f-d82d-409d-bb0b-2c4febf776f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:09.954989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd60737f-d82d-409d-bb0b-2c4febf776f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:09.955264 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd60737f-d82d-409d-bb0b-2c4febf776f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:09.955533 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd60737f-d82d-409d-bb0b-2c4febf776f7 admin admin] PATH_INFO: `/v3/projects/54aee622a161401ab8c120788299ba1a/users/e82ab92cb2c048c9a53094219076a4c9/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:09.972029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd60737f-d82d-409d-bb0b-2c4febf776f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=54aee622a161401ab8c120788299ba1a, user_id=e82ab92cb2c048c9a53094219076a4c9, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:09.974850 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd60737f-d82d-409d-bb0b-2c4febf776f7 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:10.043482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1256/2503] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:09 2022] PUT /identity/v3/projects/54aee622a161401ab8c120788299ba1a/users/e82ab92cb2c048c9a53094219076a4c9/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:10.047531 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1a279e-5090-49ab-b117-72413ba05371 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:10.058104 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1a279e-5090-49ab-b117-72413ba05371 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:10.120516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ceee37-0390-4001-becf-da5598ed7e6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:10.121135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ceee37-0390-4001-becf-da5598ed7e6c admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:10.121407 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ceee37-0390-4001-becf-da5598ed7e6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:10.121688 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ceee37-0390-4001-becf-da5598ed7e6c admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:10.122327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ceee37-0390-4001-becf-da5598ed7e6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:10.124145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ceee37-0390-4001-becf-da5598ed7e6c admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:10.137405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1248/2504] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:10 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:10.144638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd60737f-d82d-409d-bb0b-2c4febf776f7 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:10.155653 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd60737f-d82d-409d-bb0b-2c4febf776f7 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:10.161270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9c0e2f-5a7b-4820-b496-e92e1b3ecde4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:10.162999 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9c0e2f-5a7b-4820-b496-e92e1b3ecde4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:10.163281 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9c0e2f-5a7b-4820-b496-e92e1b3ecde4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:10.163932 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9c0e2f-5a7b-4820-b496-e92e1b3ecde4 admin admin] PATH_INFO: `/v3/projects/54aee622a161401ab8c120788299ba1a/users/e82ab92cb2c048c9a53094219076a4c9/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:10.167307 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9c0e2f-5a7b-4820-b496-e92e1b3ecde4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=54aee622a161401ab8c120788299ba1a, user_id=e82ab92cb2c048c9a53094219076a4c9, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:10.172394 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9c0e2f-5a7b-4820-b496-e92e1b3ecde4 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:10.184808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1257/2505] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:10 2022] PUT /identity/v3/projects/54aee622a161401ab8c120788299ba1a/users/e82ab92cb2c048c9a53094219076a4c9/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:10.196168 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192bb361-da5a-4409-895d-474179fb8621 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:10.203391 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192bb361-da5a-4409-895d-474179fb8621 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:10.203391 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192bb361-da5a-4409-895d-474179fb8621 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:10.228590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-192bb361-da5a-4409-895d-474179fb8621 None None] MFA Rules not processed for user `e82ab92cb2c048c9a53094219076a4c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:10.229801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-192bb361-da5a-4409-895d-474179fb8621 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:10.305617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1249/2506] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:10 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:10.316504 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9c0e2f-5a7b-4820-b496-e92e1b3ecde4 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:10.318209 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-4c9c0e2f-5a7b-4820-b496-e92e1b3ecde4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:10.379143 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:10.384410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9c0e2f-5a7b-4820-b496-e92e1b3ecde4 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:10.388193 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9f2f2b-a236-4efa-9751-506a173d205b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:10.394089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9f2f2b-a236-4efa-9751-506a173d205b service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:10.394089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9f2f2b-a236-4efa-9751-506a173d205b service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:10.394089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9f2f2b-a236-4efa-9751-506a173d205b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:10.399328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9f2f2b-a236-4efa-9751-506a173d205b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:10.400806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9f2f2b-a236-4efa-9751-506a173d205b service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:10.411343 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1258/2507] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:10 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:10.461524 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192bb361-da5a-4409-895d-474179fb8621 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:10.473031 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192bb361-da5a-4409-895d-474179fb8621 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:10.477049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2686ee-6a51-42a9-8cf4-eb9be670cdfb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:10.480190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2686ee-6a51-42a9-8cf4-eb9be670cdfb service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:10.480190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2686ee-6a51-42a9-8cf4-eb9be670cdfb service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:10.480190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2686ee-6a51-42a9-8cf4-eb9be670cdfb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:10.540537 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2686ee-6a51-42a9-8cf4-eb9be670cdfb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:10.542043 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2686ee-6a51-42a9-8cf4-eb9be670cdfb service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:10.552875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1250/2508] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:10 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:14.904986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9f2f2b-a236-4efa-9751-506a173d205b service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:14.906496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-8c9f2f2b-a236-4efa-9751-506a173d205b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:14.972153 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9f2f2b-a236-4efa-9751-506a173d205b service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:14.976783 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4bf15c-b7fc-4107-af76-41bb4bd344c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:14.977409 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4bf15c-b7fc-4107-af76-41bb4bd344c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:14.977681 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4bf15c-b7fc-4107-af76-41bb4bd344c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:14.980593 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4bf15c-b7fc-4107-af76-41bb4bd344c2 admin admin] PATH_INFO: `/v3/users/986d41bb84464d4588f719063fe25ff9` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:14.980593 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4bf15c-b7fc-4107-af76-41bb4bd344c2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=986d41bb84464d4588f719063fe25ff9)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:14.981240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4bf15c-b7fc-4107-af76-41bb4bd344c2 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:15.199615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ca4bf15c-b7fc-4107-af76-41bb4bd344c2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '986d41bb84464d4588f719063fe25ff9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:15.207663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ca4bf15c-b7fc-4107-af76-41bb4bd344c2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '986d41bb84464d4588f719063fe25ff9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:15.223980 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ca4bf15c-b7fc-4107-af76-41bb4bd344c2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '986d41bb84464d4588f719063fe25ff9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:15.284017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ca4bf15c-b7fc-4107-af76-41bb4bd344c2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '986d41bb84464d4588f719063fe25ff9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:15.284891 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ca4bf15c-b7fc-4107-af76-41bb4bd344c2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '986d41bb84464d4588f719063fe25ff9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:15.291496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1259/2509] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:14 2022] DELETE /identity/v3/users/986d41bb84464d4588f719063fe25ff9 => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:15.295976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c422b0-5768-4997-95e6-7fab285bdcf5 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:15.296486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c422b0-5768-4997-95e6-7fab285bdcf5 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:15.296594 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c422b0-5768-4997-95e6-7fab285bdcf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:15.315772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-34c422b0-5768-4997-95e6-7fab285bdcf5 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:15.319981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-34c422b0-5768-4997-95e6-7fab285bdcf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:15.330822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:15.382710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1251/2510] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:15.393093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4bf15c-b7fc-4107-af76-41bb4bd344c2 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:15.394651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ca4bf15c-b7fc-4107-af76-41bb4bd344c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:15.453966 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4bf15c-b7fc-4107-af76-41bb4bd344c2 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:15.458727 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b582f330-8e35-479c-9906-9d905fa78558 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:15.469026 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b582f330-8e35-479c-9906-9d905fa78558 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:15.469026 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b582f330-8e35-479c-9906-9d905fa78558 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:15.469026 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b582f330-8e35-479c-9906-9d905fa78558 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:15.469360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b582f330-8e35-479c-9906-9d905fa78558 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:15.470519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b582f330-8e35-479c-9906-9d905fa78558 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:15.480933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1260/2511] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:15.509277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c422b0-5768-4997-95e6-7fab285bdcf5 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:15.510868 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-34c422b0-5768-4997-95e6-7fab285bdcf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:15.525499 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c422b0-5768-4997-95e6-7fab285bdcf5 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:15.533084 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3ea8c9-553d-43c1-90da-4302652d6ee6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:15.533818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3ea8c9-553d-43c1-90da-4302652d6ee6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:15.534168 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3ea8c9-553d-43c1-90da-4302652d6ee6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:15.534509 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3ea8c9-553d-43c1-90da-4302652d6ee6 admin admin] PATH_INFO: `/v3/users/77368260eb8b463681ccea01dcf84afd` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:15.536322 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3ea8c9-553d-43c1-90da-4302652d6ee6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=77368260eb8b463681ccea01dcf84afd)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:15.538744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3ea8c9-553d-43c1-90da-4302652d6ee6 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:15.728858 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-3e3ea8c9-553d-43c1-90da-4302652d6ee6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '77368260eb8b463681ccea01dcf84afd'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:15.738103 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-3e3ea8c9-553d-43c1-90da-4302652d6ee6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '77368260eb8b463681ccea01dcf84afd'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:15.739294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-3e3ea8c9-553d-43c1-90da-4302652d6ee6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '77368260eb8b463681ccea01dcf84afd'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:15.755194 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-3e3ea8c9-553d-43c1-90da-4302652d6ee6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '77368260eb8b463681ccea01dcf84afd'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:15.819977 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-3e3ea8c9-553d-43c1-90da-4302652d6ee6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '77368260eb8b463681ccea01dcf84afd'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:15.826998 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1252/2512] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:15 2022] DELETE /identity/v3/users/77368260eb8b463681ccea01dcf84afd => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:15.831825 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b13c0d0-062b-485f-9cd5-133d2c2bcfe8 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:15.832159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b13c0d0-062b-485f-9cd5-133d2c2bcfe8 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:15.832444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b13c0d0-062b-485f-9cd5-133d2c2bcfe8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:15.852978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-1b13c0d0-062b-485f-9cd5-133d2c2bcfe8 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:15.854422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1b13c0d0-062b-485f-9cd5-133d2c2bcfe8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:15.860309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:15.879613 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3ea8c9-553d-43c1-90da-4302652d6ee6 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:15.881066 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-3e3ea8c9-553d-43c1-90da-4302652d6ee6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:15.927272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1261/2513] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:15.937493 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b13c0d0-062b-485f-9cd5-133d2c2bcfe8 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:15.939962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-1b13c0d0-062b-485f-9cd5-133d2c2bcfe8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:15.983904 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3ea8c9-553d-43c1-90da-4302652d6ee6 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:15.989662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971645c6-689c-484c-bb73-62a1fc2a985d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:15.990176 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971645c6-689c-484c-bb73-62a1fc2a985d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:15.990566 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971645c6-689c-484c-bb73-62a1fc2a985d admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:15.990900 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971645c6-689c-484c-bb73-62a1fc2a985d admin admin] PATH_INFO: `/v3/projects/bde084df400f4e93b631a21a0acddf08` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:15.992523 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971645c6-689c-484c-bb73-62a1fc2a985d admin admin] RBAC: Authorizing `identity:delete_project(project_id=bde084df400f4e93b631a21a0acddf08)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:15.994887 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971645c6-689c-484c-bb73-62a1fc2a985d admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:16.006497 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b13c0d0-062b-485f-9cd5-133d2c2bcfe8 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:16.010453 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddee48f-54ef-4462-9b11-2c97f7ef6f37 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:16.011074 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddee48f-54ef-4462-9b11-2c97f7ef6f37 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:16.011353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddee48f-54ef-4462-9b11-2c97f7ef6f37 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:16.011625 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddee48f-54ef-4462-9b11-2c97f7ef6f37 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:16.022240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddee48f-54ef-4462-9b11-2c97f7ef6f37 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:16.023576 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddee48f-54ef-4462-9b11-2c97f7ef6f37 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:16.044143 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1262/2514] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:16.117056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:16.127422 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-971645c6-689c-484c-bb73-62a1fc2a985d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bde084df400f4e93b631a21a0acddf08', 'request_id': 'req-971645c6-689c-484c-bb73-62a1fc2a985d'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:16.132868 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-971645c6-689c-484c-bb73-62a1fc2a985d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bde084df400f4e93b631a21a0acddf08', 'request_id': 'req-971645c6-689c-484c-bb73-62a1fc2a985d'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:16.140055 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-971645c6-689c-484c-bb73-62a1fc2a985d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bde084df400f4e93b631a21a0acddf08', 'request_id': 'req-971645c6-689c-484c-bb73-62a1fc2a985d'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:16.237755 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-971645c6-689c-484c-bb73-62a1fc2a985d admin admin] The token cache is being invalidate because project bde084df400f4e93b631a21a0acddf08 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:16.237755 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-971645c6-689c-484c-bb73-62a1fc2a985d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:16.238483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-971645c6-689c-484c-bb73-62a1fc2a985d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:16.243600 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1253/2515] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:15 2022] DELETE /identity/v3/projects/bde084df400f4e93b631a21a0acddf08 => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:16.245161 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc844014-63d7-4040-a83d-bc903c990a96 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:16.245481 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc844014-63d7-4040-a83d-bc903c990a96 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:16.245757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc844014-63d7-4040-a83d-bc903c990a96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:16.266055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-cc844014-63d7-4040-a83d-bc903c990a96 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:16.267386 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-cc844014-63d7-4040-a83d-bc903c990a96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:16.339738 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1263/2516] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:33:16.349791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba44f845-9f5d-4b91-860d-24cdde4142dd None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:16.350259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba44f845-9f5d-4b91-860d-24cdde4142dd None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:16.350360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba44f845-9f5d-4b91-860d-24cdde4142dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:16.371952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-ba44f845-9f5d-4b91-860d-24cdde4142dd None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:16.373205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ba44f845-9f5d-4b91-860d-24cdde4142dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:16.381664 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1254/2517] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:16.386058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc844014-63d7-4040-a83d-bc903c990a96 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:16.395834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc844014-63d7-4040-a83d-bc903c990a96 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:16.400494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed3569a-51e2-4866-bd10-18b52fbecb3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:16.401090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed3569a-51e2-4866-bd10-18b52fbecb3b admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:16.401367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed3569a-51e2-4866-bd10-18b52fbecb3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:16.401638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed3569a-51e2-4866-bd10-18b52fbecb3b admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:16.402280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed3569a-51e2-4866-bd10-18b52fbecb3b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:33:16.402579 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed3569a-51e2-4866-bd10-18b52fbecb3b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:16.404182 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed3569a-51e2-4866-bd10-18b52fbecb3b admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:16.415952 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1264/2518] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:33:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:33:16.420338 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba44f845-9f5d-4b91-860d-24cdde4142dd None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:16.430054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba44f845-9f5d-4b91-860d-24cdde4142dd None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:16.434716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3de276-70d9-4153-a4b5-e95b7c4ce0b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:16.435306 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3de276-70d9-4153-a4b5-e95b7c4ce0b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:16.435582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3de276-70d9-4153-a4b5-e95b7c4ce0b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:16.435850 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3de276-70d9-4153-a4b5-e95b7c4ce0b3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:16.436510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3de276-70d9-4153-a4b5-e95b7c4ce0b3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:16.438372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3de276-70d9-4153-a4b5-e95b7c4ce0b3 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:16.472758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed3569a-51e2-4866-bd10-18b52fbecb3b admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:16.474835 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-fed3569a-51e2-4866-bd10-18b52fbecb3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:16.496383 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed3569a-51e2-4866-bd10-18b52fbecb3b admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:16.499661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f691a89d-c060-4e10-8ef5-bcdae2989395 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:16.499845 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691a89d-c060-4e10-8ef5-bcdae2989395 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:16.500149 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691a89d-c060-4e10-8ef5-bcdae2989395 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:16.500428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691a89d-c060-4e10-8ef5-bcdae2989395 admin admin] PATH_INFO: `/v3/projects/995b9d4778db4364b42baa7b843025d8` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:16.501651 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f691a89d-c060-4e10-8ef5-bcdae2989395 admin admin] RBAC: Authorizing `identity:delete_project(project_id=995b9d4778db4364b42baa7b843025d8)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:16.503645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f691a89d-c060-4e10-8ef5-bcdae2989395 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:16.562906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1255/2519] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:33:16 2022] POST /identity/v3/projects => generated 403 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:16.567035 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3de276-70d9-4153-a4b5-e95b7c4ce0b3 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:16.623750 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:16.631921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3de276-70d9-4153-a4b5-e95b7c4ce0b3 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:16.634107 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f691a89d-c060-4e10-8ef5-bcdae2989395 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '995b9d4778db4364b42baa7b843025d8', 'request_id': 'req-f691a89d-c060-4e10-8ef5-bcdae2989395'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:16.687074 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b063da-3562-4ecb-9570-fec7e39298dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:16.687683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b063da-3562-4ecb-9570-fec7e39298dd admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:16.687958 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b063da-3562-4ecb-9570-fec7e39298dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:16.688258 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b063da-3562-4ecb-9570-fec7e39298dd admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:16.688908 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b063da-3562-4ecb-9570-fec7e39298dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:16.690802 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b063da-3562-4ecb-9570-fec7e39298dd admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:16.734268 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f691a89d-c060-4e10-8ef5-bcdae2989395 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '995b9d4778db4364b42baa7b843025d8', 'request_id': 'req-f691a89d-c060-4e10-8ef5-bcdae2989395'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:16.740528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f691a89d-c060-4e10-8ef5-bcdae2989395 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '995b9d4778db4364b42baa7b843025d8', 'request_id': 'req-f691a89d-c060-4e10-8ef5-bcdae2989395'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:16.753719 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f691a89d-c060-4e10-8ef5-bcdae2989395 admin admin] The token cache is being invalidate because project 995b9d4778db4364b42baa7b843025d8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:16.753949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f691a89d-c060-4e10-8ef5-bcdae2989395 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:16.754608 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f691a89d-c060-4e10-8ef5-bcdae2989395 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:16.756218 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1265/2520] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:16 2022] DELETE /identity/v3/projects/995b9d4778db4364b42baa7b843025d8 => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:16.759984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f691a89d-c060-4e10-8ef5-bcdae2989395 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:16.761342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-f691a89d-c060-4e10-8ef5-bcdae2989395 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:16.775224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f691a89d-c060-4e10-8ef5-bcdae2989395 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:16.779907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f97b55-264e-495b-a111-f9f46b1c3f1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:16.780557 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f97b55-264e-495b-a111-f9f46b1c3f1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:16.780833 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f97b55-264e-495b-a111-f9f46b1c3f1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:16.781102 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f97b55-264e-495b-a111-f9f46b1c3f1d admin admin] PATH_INFO: `/v3/users/e82ab92cb2c048c9a53094219076a4c9` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:16.782247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f97b55-264e-495b-a111-f9f46b1c3f1d admin admin] RBAC: Authorizing `identity:delete_user(user_id=e82ab92cb2c048c9a53094219076a4c9)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:16.785263 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f97b55-264e-495b-a111-f9f46b1c3f1d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:16.867926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1256/2521] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:33:16 2022] POST /identity/v3/users => generated 302 bytes in 301 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:33:16.872032 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b063da-3562-4ecb-9570-fec7e39298dd admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:16.873716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f4b063da-3562-4ecb-9570-fec7e39298dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:16.885930 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b063da-3562-4ecb-9570-fec7e39298dd admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:16.890820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb638ba-6ddf-4f84-baca-f49677e1443b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:16.891402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb638ba-6ddf-4f84-baca-f49677e1443b admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:16.891677 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb638ba-6ddf-4f84-baca-f49677e1443b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:16.892604 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb638ba-6ddf-4f84-baca-f49677e1443b admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:16.893235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb638ba-6ddf-4f84-baca-f49677e1443b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:16.894896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb638ba-6ddf-4f84-baca-f49677e1443b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:16.905130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1257/2522] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:16 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:33:16.918902 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb638ba-6ddf-4f84-baca-f49677e1443b admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:16.928652 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb638ba-6ddf-4f84-baca-f49677e1443b admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:16.933287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b387af-b11a-49c4-b386-1489e0326999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:16.933853 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b387af-b11a-49c4-b386-1489e0326999 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:16.934133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b387af-b11a-49c4-b386-1489e0326999 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:16.934400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b387af-b11a-49c4-b386-1489e0326999 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:16.935021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b387af-b11a-49c4-b386-1489e0326999 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:16.936605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b387af-b11a-49c4-b386-1489e0326999 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:16.947350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1258/2523] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:16 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:33:16.951305 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b387af-b11a-49c4-b386-1489e0326999 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:16.961640 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b387af-b11a-49c4-b386-1489e0326999 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:16.966276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fae195-5c7a-4122-af0b-0c7c3c061662 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:16.966873 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fae195-5c7a-4122-af0b-0c7c3c061662 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:16.967149 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fae195-5c7a-4122-af0b-0c7c3c061662 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:16.967416 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fae195-5c7a-4122-af0b-0c7c3c061662 admin admin] PATH_INFO: `/v3/projects/76466df5c2164b0c96ea650b2c6429f5/users/a47226f483864dd98f9e6767531d1096/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:16.970425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:DELETE=5,keystone:SELECT=90,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:16.981598 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fae195-5c7a-4122-af0b-0c7c3c061662 admin admin] RBAC: Authorizing `identity:create_grant(project_id=76466df5c2164b0c96ea650b2c6429f5, user_id=a47226f483864dd98f9e6767531d1096, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:16.984264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fae195-5c7a-4122-af0b-0c7c3c061662 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:17.005439 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e6f97b55-264e-495b-a111-f9f46b1c3f1d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e82ab92cb2c048c9a53094219076a4c9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:17.012217 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e6f97b55-264e-495b-a111-f9f46b1c3f1d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e82ab92cb2c048c9a53094219076a4c9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:17.024527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e6f97b55-264e-495b-a111-f9f46b1c3f1d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e82ab92cb2c048c9a53094219076a4c9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:17.106039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1259/2524] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:16 2022] PUT /identity/v3/projects/76466df5c2164b0c96ea650b2c6429f5/users/a47226f483864dd98f9e6767531d1096/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:33:17.116300 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47839c6-46e3-4e15-a166-da72e12fc9b4 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:17.116578 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47839c6-46e3-4e15-a166-da72e12fc9b4 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:17.116846 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47839c6-46e3-4e15-a166-da72e12fc9b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:17.130282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e6f97b55-264e-495b-a111-f9f46b1c3f1d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e82ab92cb2c048c9a53094219076a4c9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:17.130602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e6f97b55-264e-495b-a111-f9f46b1c3f1d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e82ab92cb2c048c9a53094219076a4c9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:17.138828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1266/2525] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:16 2022] DELETE /identity/v3/users/e82ab92cb2c048c9a53094219076a4c9 => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:33:17.148233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f47839c6-46e3-4e15-a166-da72e12fc9b4 None None] MFA Rules not processed for user `a47226f483864dd98f9e6767531d1096`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:17.149468 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f47839c6-46e3-4e15-a166-da72e12fc9b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:17.233703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1260/2526] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:17 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:33:17.239194 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f97b55-264e-495b-a111-f9f46b1c3f1d admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:17.240843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e6f97b55-264e-495b-a111-f9f46b1c3f1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:17.310124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f97b55-264e-495b-a111-f9f46b1c3f1d admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:17.315319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b5069e-6ff2-4de2-8b03-59e761a77e62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:17.320140 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b5069e-6ff2-4de2-8b03-59e761a77e62 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:17.320140 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b5069e-6ff2-4de2-8b03-59e761a77e62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:17.320140 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b5069e-6ff2-4de2-8b03-59e761a77e62 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:17.320140 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b5069e-6ff2-4de2-8b03-59e761a77e62 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:17.320140 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b5069e-6ff2-4de2-8b03-59e761a77e62 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:17.435822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1267/2527] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:33:17 2022] POST /identity/v3/projects => generated 405 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:17.440478 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47839c6-46e3-4e15-a166-da72e12fc9b4 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:17.456877 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47839c6-46e3-4e15-a166-da72e12fc9b4 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:17.531998 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adea1b1-58b5-4d62-bc30-ff005cb03cc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:17.532740 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adea1b1-58b5-4d62-bc30-ff005cb03cc4 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:17.533015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adea1b1-58b5-4d62-bc30-ff005cb03cc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:17.533285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adea1b1-58b5-4d62-bc30-ff005cb03cc4 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:17.533912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adea1b1-58b5-4d62-bc30-ff005cb03cc4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:17.535886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adea1b1-58b5-4d62-bc30-ff005cb03cc4 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:17.583100 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b5069e-6ff2-4de2-8b03-59e761a77e62 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:17.585772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-07b5069e-6ff2-4de2-8b03-59e761a77e62 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:17.620455 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b5069e-6ff2-4de2-8b03-59e761a77e62 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:17.631497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1261/2528] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:33:17 2022] POST /identity/v3/users => generated 309 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:17.633462 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adea1b1-58b5-4d62-bc30-ff005cb03cc4 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:17.633641 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0efefc-af40-4815-b1d9-c19e45f20bbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:17.636481 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0efefc-af40-4815-b1d9-c19e45f20bbe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:17.637015 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0efefc-af40-4815-b1d9-c19e45f20bbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:17.637569 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0efefc-af40-4815-b1d9-c19e45f20bbe admin admin] PATH_INFO: `/v3/projects/54aee622a161401ab8c120788299ba1a` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:17.639582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0efefc-af40-4815-b1d9-c19e45f20bbe admin admin] RBAC: Authorizing `identity:delete_project(project_id=54aee622a161401ab8c120788299ba1a)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:17.641627 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0efefc-af40-4815-b1d9-c19e45f20bbe admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:17.646129 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adea1b1-58b5-4d62-bc30-ff005cb03cc4 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:17.652252 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5325f145-3b0b-4b74-b445-5dee458a46d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:17.653250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5325f145-3b0b-4b74-b445-5dee458a46d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:17.653250 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5325f145-3b0b-4b74-b445-5dee458a46d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:17.653457 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5325f145-3b0b-4b74-b445-5dee458a46d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:17.654167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5325f145-3b0b-4b74-b445-5dee458a46d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:17.656453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5325f145-3b0b-4b74-b445-5dee458a46d9 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:17.662353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:17.671212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1262/2529] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:17 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:33:17.672421 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5325f145-3b0b-4b74-b445-5dee458a46d9 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:17.683895 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5325f145-3b0b-4b74-b445-5dee458a46d9 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:17.689986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5d87ad-f528-45d9-95be-ec9912d5604f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:17.691258 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5d87ad-f528-45d9-95be-ec9912d5604f admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:17.691258 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5d87ad-f528-45d9-95be-ec9912d5604f admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:17.691258 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5d87ad-f528-45d9-95be-ec9912d5604f admin admin] PATH_INFO: `/v3/projects/20e6489c6810450592eff05afd94c333/users/d53715ba96b04fb29dadde049a0fae92/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:17.707612 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5d87ad-f528-45d9-95be-ec9912d5604f admin admin] RBAC: Authorizing `identity:create_grant(project_id=20e6489c6810450592eff05afd94c333, user_id=d53715ba96b04fb29dadde049a0fae92, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:17.710535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5d87ad-f528-45d9-95be-ec9912d5604f admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:17.838213 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1263/2530] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:17 2022] PUT /identity/v3/projects/20e6489c6810450592eff05afd94c333/users/d53715ba96b04fb29dadde049a0fae92/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:33:17.841973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5d87ad-f528-45d9-95be-ec9912d5604f admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:17.846466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3f0efefc-af40-4815-b1d9-c19e45f20bbe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '54aee622a161401ab8c120788299ba1a', 'request_id': 'req-3f0efefc-af40-4815-b1d9-c19e45f20bbe'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:17.874219 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:17.907026 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5d87ad-f528-45d9-95be-ec9912d5604f admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:17.964751 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a76896-0298-4af4-8be2-d80f82cdc56b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:17.965368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a76896-0298-4af4-8be2-d80f82cdc56b admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:17.965643 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a76896-0298-4af4-8be2-d80f82cdc56b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:17.965912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a76896-0298-4af4-8be2-d80f82cdc56b admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:17.966616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a76896-0298-4af4-8be2-d80f82cdc56b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:17.968243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a76896-0298-4af4-8be2-d80f82cdc56b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:17.979361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1264/2531] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:17 2022] GET /identity/v3/roles => generated 1420 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:33:17.983489 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a76896-0298-4af4-8be2-d80f82cdc56b admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:17.991507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3f0efefc-af40-4815-b1d9-c19e45f20bbe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '54aee622a161401ab8c120788299ba1a', 'request_id': 'req-3f0efefc-af40-4815-b1d9-c19e45f20bbe'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:17.995819 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a76896-0298-4af4-8be2-d80f82cdc56b admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:17.998260 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3f0efefc-af40-4815-b1d9-c19e45f20bbe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '54aee622a161401ab8c120788299ba1a', 'request_id': 'req-3f0efefc-af40-4815-b1d9-c19e45f20bbe'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:17.999259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4394587-05c8-4192-b1bd-d962bac5b5e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:17.999921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4394587-05c8-4192-b1bd-d962bac5b5e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:18.000229 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4394587-05c8-4192-b1bd-d962bac5b5e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:18.000500 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4394587-05c8-4192-b1bd-d962bac5b5e7 admin admin] PATH_INFO: `/v3/projects/20e6489c6810450592eff05afd94c333/users/d53715ba96b04fb29dadde049a0fae92/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:18.002472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4394587-05c8-4192-b1bd-d962bac5b5e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=20e6489c6810450592eff05afd94c333, user_id=d53715ba96b04fb29dadde049a0fae92, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:18.005201 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4394587-05c8-4192-b1bd-d962bac5b5e7 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:18.011281 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3f0efefc-af40-4815-b1d9-c19e45f20bbe admin admin] The token cache is being invalidate because project 54aee622a161401ab8c120788299ba1a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:18.011824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3f0efefc-af40-4815-b1d9-c19e45f20bbe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:18.012321 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-3f0efefc-af40-4815-b1d9-c19e45f20bbe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:18.014201 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1268/2532] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:17 2022] DELETE /identity/v3/projects/54aee622a161401ab8c120788299ba1a => generated 0 bytes in 432 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:18.018194 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1265/2533] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:17 2022] PUT /identity/v3/projects/20e6489c6810450592eff05afd94c333/users/d53715ba96b04fb29dadde049a0fae92/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:33:18.021163 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4f09cf-f1e8-4d01-b4bb-ce59a1005a48 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:18.029963 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4f09cf-f1e8-4d01-b4bb-ce59a1005a48 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:18.029963 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4f09cf-f1e8-4d01-b4bb-ce59a1005a48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:18.039909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626103df-3108-40cc-a5ec-6c5028a7ebe7 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:18.040227 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626103df-3108-40cc-a5ec-6c5028a7ebe7 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:18.040515 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626103df-3108-40cc-a5ec-6c5028a7ebe7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:18.041466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-5c4f09cf-f1e8-4d01-b4bb-ce59a1005a48 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:18.042715 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-5c4f09cf-f1e8-4d01-b4bb-ce59a1005a48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:18.068424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-626103df-3108-40cc-a5ec-6c5028a7ebe7 None None] MFA Rules not processed for user `d53715ba96b04fb29dadde049a0fae92`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:18.069633 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-626103df-3108-40cc-a5ec-6c5028a7ebe7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:18.103181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1269/2534] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:18.110864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a872b0a8-aefe-44fc-aca6-bd398f7fdfcc None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:18.111128 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a872b0a8-aefe-44fc-aca6-bd398f7fdfcc None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:18.111401 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a872b0a8-aefe-44fc-aca6-bd398f7fdfcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:18.132852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-a872b0a8-aefe-44fc-aca6-bd398f7fdfcc None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:18.133398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-a872b0a8-aefe-44fc-aca6-bd398f7fdfcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:18.143204 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1270/2535] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:33:18.147433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a872b0a8-aefe-44fc-aca6-bd398f7fdfcc None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:18.153402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1266/2536] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:18 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:18.158344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a872b0a8-aefe-44fc-aca6-bd398f7fdfcc None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:18.164398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814748b1-5356-4fa2-a25c-473e71145762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:18.164398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814748b1-5356-4fa2-a25c-473e71145762 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:18.164398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814748b1-5356-4fa2-a25c-473e71145762 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:18.164896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814748b1-5356-4fa2-a25c-473e71145762 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:18.165047 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626103df-3108-40cc-a5ec-6c5028a7ebe7 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:18.165698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814748b1-5356-4fa2-a25c-473e71145762 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:33:18.166018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814748b1-5356-4fa2-a25c-473e71145762 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:18.166570 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-626103df-3108-40cc-a5ec-6c5028a7ebe7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:18.167757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814748b1-5356-4fa2-a25c-473e71145762 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:18.185399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1271/2537] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:33:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 21 01:33:18.187573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814748b1-5356-4fa2-a25c-473e71145762 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:18.200552 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814748b1-5356-4fa2-a25c-473e71145762 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:18.206327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a233781-d3b9-4b5a-bf5f-3dafe7d1002d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:18.206943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a233781-d3b9-4b5a-bf5f-3dafe7d1002d admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:18.207217 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a233781-d3b9-4b5a-bf5f-3dafe7d1002d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:18.207486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a233781-d3b9-4b5a-bf5f-3dafe7d1002d admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:18.208126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a233781-d3b9-4b5a-bf5f-3dafe7d1002d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:18.210026 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a233781-d3b9-4b5a-bf5f-3dafe7d1002d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:18.219570 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:18.231311 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626103df-3108-40cc-a5ec-6c5028a7ebe7 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:18.235218 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7ca1ed-ccb2-4c09-bbab-bef272b92791 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:18.235857 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7ca1ed-ccb2-4c09-bbab-bef272b92791 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:18.236168 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7ca1ed-ccb2-4c09-bbab-bef272b92791 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:18.236444 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7ca1ed-ccb2-4c09-bbab-bef272b92791 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:18.245451 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7ca1ed-ccb2-4c09-bbab-bef272b92791 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:18.246874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7ca1ed-ccb2-4c09-bbab-bef272b92791 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:18.256566 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1267/2538] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:18 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:33:18.313890 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1272/2539] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:33:18 2022] POST /identity/v3/projects => generated 409 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:18.318219 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7ca1ed-ccb2-4c09-bbab-bef272b92791 service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:18.329958 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7ca1ed-ccb2-4c09-bbab-bef272b92791 service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:18.396578 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17787f2b-2369-4d4c-b781-f1865709dd74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:18.397280 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17787f2b-2369-4d4c-b781-f1865709dd74 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:18.397623 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17787f2b-2369-4d4c-b781-f1865709dd74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:18.397964 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17787f2b-2369-4d4c-b781-f1865709dd74 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:18.398645 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17787f2b-2369-4d4c-b781-f1865709dd74 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:18.401087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17787f2b-2369-4d4c-b781-f1865709dd74 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:18.421354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a233781-d3b9-4b5a-bf5f-3dafe7d1002d admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:18.423217 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6a233781-d3b9-4b5a-bf5f-3dafe7d1002d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:18.436544 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a233781-d3b9-4b5a-bf5f-3dafe7d1002d admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:18.442182 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7248e6b-a900-45cf-bb9c-aaf9b6725083 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:18.442844 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7248e6b-a900-45cf-bb9c-aaf9b6725083 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:18.443119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7248e6b-a900-45cf-bb9c-aaf9b6725083 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:18.443385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7248e6b-a900-45cf-bb9c-aaf9b6725083 admin admin] PATH_INFO: `/v3/users/a47226f483864dd98f9e6767531d1096` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:18.444594 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7248e6b-a900-45cf-bb9c-aaf9b6725083 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a47226f483864dd98f9e6767531d1096)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:18.446615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7248e6b-a900-45cf-bb9c-aaf9b6725083 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:18.481756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1268/2540] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:33:18 2022] POST /identity/v3/users => generated 305 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:18.485994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17787f2b-2369-4d4c-b781-f1865709dd74 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:18.499931 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17787f2b-2369-4d4c-b781-f1865709dd74 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:18.505430 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ec741f-27c9-4394-a8b2-d7da7277f9f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:18.508113 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ec741f-27c9-4394-a8b2-d7da7277f9f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:18.508113 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ec741f-27c9-4394-a8b2-d7da7277f9f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:18.508113 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ec741f-27c9-4394-a8b2-d7da7277f9f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:18.508113 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ec741f-27c9-4394-a8b2-d7da7277f9f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:18.508820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ec741f-27c9-4394-a8b2-d7da7277f9f6 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:18.519986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1269/2541] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:18 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:33:18.524216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ec741f-27c9-4394-a8b2-d7da7277f9f6 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:18.534466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ec741f-27c9-4394-a8b2-d7da7277f9f6 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:18.539278 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af04f24-e932-4ccc-a6df-1ff990972060 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:18.542486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af04f24-e932-4ccc-a6df-1ff990972060 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:18.542486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af04f24-e932-4ccc-a6df-1ff990972060 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:18.542486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af04f24-e932-4ccc-a6df-1ff990972060 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:18.542486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af04f24-e932-4ccc-a6df-1ff990972060 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:18.542864 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af04f24-e932-4ccc-a6df-1ff990972060 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:18.553784 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1270/2542] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:18 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:33:18.557761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af04f24-e932-4ccc-a6df-1ff990972060 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:18.568483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af04f24-e932-4ccc-a6df-1ff990972060 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:18.573231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b328802e-07b9-41a3-942e-a3cb61c342d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:18.573885 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b328802e-07b9-41a3-942e-a3cb61c342d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:18.574159 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b328802e-07b9-41a3-942e-a3cb61c342d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:18.574427 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b328802e-07b9-41a3-942e-a3cb61c342d5 admin admin] PATH_INFO: `/v3/projects/909a4c4d763b464680c5beda9eaea6dc/users/3f6fcc97f82e481581f60f936ad3e4fa/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:18.590713 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b328802e-07b9-41a3-942e-a3cb61c342d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=909a4c4d763b464680c5beda9eaea6dc, user_id=3f6fcc97f82e481581f60f936ad3e4fa, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:18.593493 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b328802e-07b9-41a3-942e-a3cb61c342d5 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:18.636639 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f7248e6b-a900-45cf-bb9c-aaf9b6725083 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a47226f483864dd98f9e6767531d1096'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:18.645575 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1271/2543] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:18 2022] PUT /identity/v3/projects/909a4c4d763b464680c5beda9eaea6dc/users/3f6fcc97f82e481581f60f936ad3e4fa/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:33:18.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f7248e6b-a900-45cf-bb9c-aaf9b6725083 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a47226f483864dd98f9e6767531d1096'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:18.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:18.663351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f7248e6b-a900-45cf-bb9c-aaf9b6725083 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a47226f483864dd98f9e6767531d1096'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:18.713175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f7248e6b-a900-45cf-bb9c-aaf9b6725083 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a47226f483864dd98f9e6767531d1096'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:18.715089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f7248e6b-a900-45cf-bb9c-aaf9b6725083 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a47226f483864dd98f9e6767531d1096'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:18.715238 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5206b9ad-f8a9-4615-b836-3b888cc23f8f None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:18.715344 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5206b9ad-f8a9-4615-b836-3b888cc23f8f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:18.715612 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5206b9ad-f8a9-4615-b836-3b888cc23f8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:18.723435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1273/2544] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:18 2022] DELETE /identity/v3/users/a47226f483864dd98f9e6767531d1096 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:33:18.728728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547f8f28-b620-44f8-81db-000d8c6c12f8 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:18.729251 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547f8f28-b620-44f8-81db-000d8c6c12f8 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:18.729600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547f8f28-b620-44f8-81db-000d8c6c12f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:18.747914 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-5206b9ad-f8a9-4615-b836-3b888cc23f8f None None] MFA Rules not processed for user `3f6fcc97f82e481581f60f936ad3e4fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:18.749326 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5206b9ad-f8a9-4615-b836-3b888cc23f8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:18.754432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-547f8f28-b620-44f8-81db-000d8c6c12f8 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:18.755635 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-547f8f28-b620-44f8-81db-000d8c6c12f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:18.818947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1274/2545] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:18.823470 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1272/2546] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:18 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:33:18.824744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547f8f28-b620-44f8-81db-000d8c6c12f8 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:18.826119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-547f8f28-b620-44f8-81db-000d8c6c12f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:18.829076 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5206b9ad-f8a9-4615-b836-3b888cc23f8f None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:18.830446 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5206b9ad-f8a9-4615-b836-3b888cc23f8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:18.838761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547f8f28-b620-44f8-81db-000d8c6c12f8 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:18.843463 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1c652c-2e27-4040-952e-14601023c5b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:18.844066 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1c652c-2e27-4040-952e-14601023c5b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:18.844383 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1c652c-2e27-4040-952e-14601023c5b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:18.844683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1c652c-2e27-4040-952e-14601023c5b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:18.845341 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1c652c-2e27-4040-952e-14601023c5b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:18.847242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1c652c-2e27-4040-952e-14601023c5b1 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:18.852479 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:18.889635 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5206b9ad-f8a9-4615-b836-3b888cc23f8f None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:18.893874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7380d2e-3638-4ec4-90a8-b6f869d41942 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:18.895840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7380d2e-3638-4ec4-90a8-b6f869d41942 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:18.895840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7380d2e-3638-4ec4-90a8-b6f869d41942 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:18.895840 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7380d2e-3638-4ec4-90a8-b6f869d41942 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:18.904249 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7380d2e-3638-4ec4-90a8-b6f869d41942 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:18.905658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7380d2e-3638-4ec4-90a8-b6f869d41942 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:18.915168 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1273/2547] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:33:18.946779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1275/2548] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:33:18 2022] POST /identity/v3/projects => generated 409 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:18.950996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7380d2e-3638-4ec4-90a8-b6f869d41942 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:18.960996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7380d2e-3638-4ec4-90a8-b6f869d41942 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:19.019906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4c6684-bc1c-4964-8d7f-c0830a7f08cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:19.020575 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4c6684-bc1c-4964-8d7f-c0830a7f08cd admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:19.020849 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4c6684-bc1c-4964-8d7f-c0830a7f08cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:19.021118 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4c6684-bc1c-4964-8d7f-c0830a7f08cd admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:19.021728 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4c6684-bc1c-4964-8d7f-c0830a7f08cd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:19.023948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4c6684-bc1c-4964-8d7f-c0830a7f08cd admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:19.103976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1274/2549] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:33:18 2022] POST /identity/v3/users => generated 311 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:33:19.105344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1c652c-2e27-4040-952e-14601023c5b1 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:19.116991 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1c652c-2e27-4040-952e-14601023c5b1 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:19.122656 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a50da9f-75d7-4809-97cb-76ddeb709318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:19.123358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a50da9f-75d7-4809-97cb-76ddeb709318 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:19.123695 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a50da9f-75d7-4809-97cb-76ddeb709318 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:19.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a50da9f-75d7-4809-97cb-76ddeb709318 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:19.124918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a50da9f-75d7-4809-97cb-76ddeb709318 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:19.126528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a50da9f-75d7-4809-97cb-76ddeb709318 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:19.138013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1276/2550] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:19 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:33:19.145856 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4c6684-bc1c-4964-8d7f-c0830a7f08cd admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:19.152036 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4c6684-bc1c-4964-8d7f-c0830a7f08cd admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:19.156681 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4272d238-de8d-460b-9b1c-36495bd381ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:19.157378 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4272d238-de8d-460b-9b1c-36495bd381ab admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:19.158145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4272d238-de8d-460b-9b1c-36495bd381ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:19.158412 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4272d238-de8d-460b-9b1c-36495bd381ab admin admin] PATH_INFO: `/v3/projects/9e6609eb6e0f4fe6904798711cd81275/users/48f25c002e2e4a32aaf18bb7c0685ad6/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:19.174371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4272d238-de8d-460b-9b1c-36495bd381ab admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e6609eb6e0f4fe6904798711cd81275, user_id=48f25c002e2e4a32aaf18bb7c0685ad6, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:19.177114 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4272d238-de8d-460b-9b1c-36495bd381ab admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:19.238294 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1275/2551] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:19 2022] PUT /identity/v3/projects/9e6609eb6e0f4fe6904798711cd81275/users/48f25c002e2e4a32aaf18bb7c0685ad6/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:19.242340 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a50da9f-75d7-4809-97cb-76ddeb709318 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:19.252285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a50da9f-75d7-4809-97cb-76ddeb709318 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:19.284966 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4272d238-de8d-460b-9b1c-36495bd381ab admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:19.287083 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-4272d238-de8d-460b-9b1c-36495bd381ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:19.316757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9717567-8f69-4d29-8d34-2f9660ef4b2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:19.316757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9717567-8f69-4d29-8d34-2f9660ef4b2d admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:19.316757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9717567-8f69-4d29-8d34-2f9660ef4b2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:19.316757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9717567-8f69-4d29-8d34-2f9660ef4b2d admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:19.317491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9717567-8f69-4d29-8d34-2f9660ef4b2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:19.317491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9717567-8f69-4d29-8d34-2f9660ef4b2d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:19.323238 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1277/2552] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:19 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:19.327530 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9717567-8f69-4d29-8d34-2f9660ef4b2d admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:19.338649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9717567-8f69-4d29-8d34-2f9660ef4b2d admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:19.346062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4702caae-b303-450c-9002-1d886e2bb67c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:19.346757 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4702caae-b303-450c-9002-1d886e2bb67c admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:19.347029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4702caae-b303-450c-9002-1d886e2bb67c admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:19.347294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4702caae-b303-450c-9002-1d886e2bb67c admin admin] PATH_INFO: `/v3/projects/9e6609eb6e0f4fe6904798711cd81275/users/48f25c002e2e4a32aaf18bb7c0685ad6/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:19.349836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4702caae-b303-450c-9002-1d886e2bb67c admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e6609eb6e0f4fe6904798711cd81275, user_id=48f25c002e2e4a32aaf18bb7c0685ad6, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:19.352619 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4702caae-b303-450c-9002-1d886e2bb67c admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:19.365256 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1278/2553] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:19 2022] PUT /identity/v3/projects/9e6609eb6e0f4fe6904798711cd81275/users/48f25c002e2e4a32aaf18bb7c0685ad6/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:19.376473 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597cb727-02a1-4f8a-8a59-c4916925083e None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:19.376761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597cb727-02a1-4f8a-8a59-c4916925083e None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:19.377033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597cb727-02a1-4f8a-8a59-c4916925083e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:19.378300 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4272d238-de8d-460b-9b1c-36495bd381ab admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:19.404329 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-597cb727-02a1-4f8a-8a59-c4916925083e None None] MFA Rules not processed for user `48f25c002e2e4a32aaf18bb7c0685ad6`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:19.405482 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-597cb727-02a1-4f8a-8a59-c4916925083e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:19.409853 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:19.418430 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=94,keystone:DELETE=3,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:19.434315 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8daef4-b879-4276-91ef-bc187bb10b88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:19.434944 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8daef4-b879-4276-91ef-bc187bb10b88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:19.435220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8daef4-b879-4276-91ef-bc187bb10b88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:19.435516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8daef4-b879-4276-91ef-bc187bb10b88 admin admin] PATH_INFO: `/v3/projects/76466df5c2164b0c96ea650b2c6429f5` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:19.436900 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8daef4-b879-4276-91ef-bc187bb10b88 admin admin] RBAC: Authorizing `identity:delete_project(project_id=76466df5c2164b0c96ea650b2c6429f5)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:19.439286 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8daef4-b879-4276-91ef-bc187bb10b88 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:19.474496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1279/2554] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:19 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:33:19.485223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597cb727-02a1-4f8a-8a59-c4916925083e None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:19.487310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-597cb727-02a1-4f8a-8a59-c4916925083e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:19.547782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1b8daef4-b879-4276-91ef-bc187bb10b88 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '76466df5c2164b0c96ea650b2c6429f5', 'request_id': 'req-1b8daef4-b879-4276-91ef-bc187bb10b88'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:19.549157 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597cb727-02a1-4f8a-8a59-c4916925083e None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:19.553830 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1b8daef4-b879-4276-91ef-bc187bb10b88 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '76466df5c2164b0c96ea650b2c6429f5', 'request_id': 'req-1b8daef4-b879-4276-91ef-bc187bb10b88'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:19.560727 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1b8daef4-b879-4276-91ef-bc187bb10b88 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '76466df5c2164b0c96ea650b2c6429f5', 'request_id': 'req-1b8daef4-b879-4276-91ef-bc187bb10b88'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:19.598769 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ecd307-21fe-4879-8e9a-f7287239218d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:19.599373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ecd307-21fe-4879-8e9a-f7287239218d service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:19.599649 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ecd307-21fe-4879-8e9a-f7287239218d service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:19.599919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ecd307-21fe-4879-8e9a-f7287239218d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:19.658959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ecd307-21fe-4879-8e9a-f7287239218d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:19.660357 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ecd307-21fe-4879-8e9a-f7287239218d service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:19.660672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1b8daef4-b879-4276-91ef-bc187bb10b88 admin admin] The token cache is being invalidate because project 76466df5c2164b0c96ea650b2c6429f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:19.660964 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1b8daef4-b879-4276-91ef-bc187bb10b88 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:19.661884 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1b8daef4-b879-4276-91ef-bc187bb10b88 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:19.663600 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1276/2555] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:19 2022] DELETE /identity/v3/projects/76466df5c2164b0c96ea650b2c6429f5 => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:33:19.667277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8daef4-b879-4276-91ef-bc187bb10b88 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:19.671008 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-1b8daef4-b879-4276-91ef-bc187bb10b88 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:19.671136 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1280/2556] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:19 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:33:19.736960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8daef4-b879-4276-91ef-bc187bb10b88 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:19.741656 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ac3896-16a5-4614-94fb-2b828f58e02a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:19.742275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ac3896-16a5-4614-94fb-2b828f58e02a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:19.742549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ac3896-16a5-4614-94fb-2b828f58e02a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:19.742823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ac3896-16a5-4614-94fb-2b828f58e02a admin admin] PATH_INFO: `/v3/users/d53715ba96b04fb29dadde049a0fae92` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:19.744061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ac3896-16a5-4614-94fb-2b828f58e02a admin admin] RBAC: Authorizing `identity:delete_user(user_id=d53715ba96b04fb29dadde049a0fae92)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:19.745994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ac3896-16a5-4614-94fb-2b828f58e02a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:19.965410 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-82ac3896-16a5-4614-94fb-2b828f58e02a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd53715ba96b04fb29dadde049a0fae92'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:19.972292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-82ac3896-16a5-4614-94fb-2b828f58e02a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd53715ba96b04fb29dadde049a0fae92'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:19.972922 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-82ac3896-16a5-4614-94fb-2b828f58e02a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd53715ba96b04fb29dadde049a0fae92'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:19.980777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:19.985304 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-82ac3896-16a5-4614-94fb-2b828f58e02a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd53715ba96b04fb29dadde049a0fae92'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:20.053595 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ecd307-21fe-4879-8e9a-f7287239218d service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:20.062222 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ecd307-21fe-4879-8e9a-f7287239218d service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:20.065312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-82ac3896-16a5-4614-94fb-2b828f58e02a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd53715ba96b04fb29dadde049a0fae92'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:20.066824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe33f1d-91da-45d6-b7c2-fb419a55fe0e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:20.067502 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe33f1d-91da-45d6-b7c2-fb419a55fe0e service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:20.067782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe33f1d-91da-45d6-b7c2-fb419a55fe0e service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:20.068067 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe33f1d-91da-45d6-b7c2-fb419a55fe0e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:20.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ffe33f1d-91da-45d6-b7c2-fb419a55fe0e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:20.072133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1277/2557] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:19 2022] DELETE /identity/v3/users/d53715ba96b04fb29dadde049a0fae92 => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:20.085756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:20.133192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe33f1d-91da-45d6-b7c2-fb419a55fe0e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:20.134632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe33f1d-91da-45d6-b7c2-fb419a55fe0e service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:20.144404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1281/2558] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:20 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:33:20.473445 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ac3896-16a5-4614-94fb-2b828f58e02a admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:20.474865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-82ac3896-16a5-4614-94fb-2b828f58e02a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:20.543274 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ac3896-16a5-4614-94fb-2b828f58e02a admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:20.547863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc800e7-aa60-4e06-9c0a-1ac2eea2428f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:20.548986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc800e7-aa60-4e06-9c0a-1ac2eea2428f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:20.549262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc800e7-aa60-4e06-9c0a-1ac2eea2428f admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:20.549526 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc800e7-aa60-4e06-9c0a-1ac2eea2428f admin admin] PATH_INFO: `/v3/projects/20e6489c6810450592eff05afd94c333` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:20.550660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc800e7-aa60-4e06-9c0a-1ac2eea2428f admin admin] RBAC: Authorizing `identity:delete_project(project_id=20e6489c6810450592eff05afd94c333)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:20.552574 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc800e7-aa60-4e06-9c0a-1ac2eea2428f admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:20.660622 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0bc800e7-aa60-4e06-9c0a-1ac2eea2428f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '20e6489c6810450592eff05afd94c333', 'request_id': 'req-0bc800e7-aa60-4e06-9c0a-1ac2eea2428f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:20.666099 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0bc800e7-aa60-4e06-9c0a-1ac2eea2428f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '20e6489c6810450592eff05afd94c333', 'request_id': 'req-0bc800e7-aa60-4e06-9c0a-1ac2eea2428f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:20.672671 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0bc800e7-aa60-4e06-9c0a-1ac2eea2428f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '20e6489c6810450592eff05afd94c333', 'request_id': 'req-0bc800e7-aa60-4e06-9c0a-1ac2eea2428f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:20.744620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0bc800e7-aa60-4e06-9c0a-1ac2eea2428f admin admin] The token cache is being invalidate because project 20e6489c6810450592eff05afd94c333 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:20.745132 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0bc800e7-aa60-4e06-9c0a-1ac2eea2428f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:20.746008 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-0bc800e7-aa60-4e06-9c0a-1ac2eea2428f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:20.748264 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1278/2559] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:20 2022] DELETE /identity/v3/projects/20e6489c6810450592eff05afd94c333 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:20.754601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c3d677-63a6-456f-8b5f-7f802fada9ab None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:20.754904 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c3d677-63a6-456f-8b5f-7f802fada9ab None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:20.755177 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c3d677-63a6-456f-8b5f-7f802fada9ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:20.775350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-94c3d677-63a6-456f-8b5f-7f802fada9ab None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:20.776669 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-94c3d677-63a6-456f-8b5f-7f802fada9ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:20.838717 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1282/2560] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:20.848134 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291a6996-b282-400d-b8be-4754dcb8e9a8 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:20.848550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291a6996-b282-400d-b8be-4754dcb8e9a8 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:20.848664 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291a6996-b282-400d-b8be-4754dcb8e9a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:20.870027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-291a6996-b282-400d-b8be-4754dcb8e9a8 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:20.871246 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-291a6996-b282-400d-b8be-4754dcb8e9a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:20.880969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1279/2561] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:20.885970 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c3d677-63a6-456f-8b5f-7f802fada9ab None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:20.896414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c3d677-63a6-456f-8b5f-7f802fada9ab None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:20.901254 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408aae52-2064-43db-a0dc-487a8fe94812 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:20.901854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408aae52-2064-43db-a0dc-487a8fe94812 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:20.902133 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408aae52-2064-43db-a0dc-487a8fe94812 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:20.902408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408aae52-2064-43db-a0dc-487a8fe94812 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:20.903050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408aae52-2064-43db-a0dc-487a8fe94812 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:33:20.903351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408aae52-2064-43db-a0dc-487a8fe94812 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:20.905023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408aae52-2064-43db-a0dc-487a8fe94812 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:20.917526 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1283/2562] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:33:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:33:20.921938 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291a6996-b282-400d-b8be-4754dcb8e9a8 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:20.932009 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291a6996-b282-400d-b8be-4754dcb8e9a8 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:20.936945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafa948e-1f5c-4777-b7c2-59c4daa4a33f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:20.937536 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafa948e-1f5c-4777-b7c2-59c4daa4a33f admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:20.937813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafa948e-1f5c-4777-b7c2-59c4daa4a33f admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:20.938095 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafa948e-1f5c-4777-b7c2-59c4daa4a33f admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:20.938716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafa948e-1f5c-4777-b7c2-59c4daa4a33f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:20.940698 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafa948e-1f5c-4777-b7c2-59c4daa4a33f admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:21.029284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1280/2563] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:33:20 2022] POST /identity/v3/projects => generated 357 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:21.031214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408aae52-2064-43db-a0dc-487a8fe94812 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:21.042425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408aae52-2064-43db-a0dc-487a8fe94812 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:21.101060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c73ece-3b35-4ed6-8a96-be111a96c2c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:21.101671 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c73ece-3b35-4ed6-8a96-be111a96c2c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:21.101948 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c73ece-3b35-4ed6-8a96-be111a96c2c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:21.102219 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c73ece-3b35-4ed6-8a96-be111a96c2c2 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:21.102860 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c73ece-3b35-4ed6-8a96-be111a96c2c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:21.104834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c73ece-3b35-4ed6-8a96-be111a96c2c2 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:21.180906 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1284/2564] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:33:21 2022] POST /identity/v3/users => generated 279 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:21.185301 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafa948e-1f5c-4777-b7c2-59c4daa4a33f admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:21.196418 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafa948e-1f5c-4777-b7c2-59c4daa4a33f admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:21.201215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6875cb9-f698-4c21-9e63-ba1a4bdecd5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:21.201823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6875cb9-f698-4c21-9e63-ba1a4bdecd5e admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:21.202095 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6875cb9-f698-4c21-9e63-ba1a4bdecd5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:21.202362 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6875cb9-f698-4c21-9e63-ba1a4bdecd5e admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:21.203027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6875cb9-f698-4c21-9e63-ba1a4bdecd5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:21.205159 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6875cb9-f698-4c21-9e63-ba1a4bdecd5e admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:21.215821 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1281/2565] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:21.220766 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c73ece-3b35-4ed6-8a96-be111a96c2c2 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:21.232060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c73ece-3b35-4ed6-8a96-be111a96c2c2 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:21.236914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b400c916-6e80-4889-83f6-602f9e1490bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:21.237516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b400c916-6e80-4889-83f6-602f9e1490bd admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:21.237787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b400c916-6e80-4889-83f6-602f9e1490bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:21.238053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b400c916-6e80-4889-83f6-602f9e1490bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:21.238688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b400c916-6e80-4889-83f6-602f9e1490bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:21.240321 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b400c916-6e80-4889-83f6-602f9e1490bd admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:21.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1285/2566] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:21 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:21.255742 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6875cb9-f698-4c21-9e63-ba1a4bdecd5e admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:21.266190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6875cb9-f698-4c21-9e63-ba1a4bdecd5e admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:21.271408 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5836fdc3-f30b-4031-b367-5060808e6225 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:21.272064 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5836fdc3-f30b-4031-b367-5060808e6225 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:21.272367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5836fdc3-f30b-4031-b367-5060808e6225 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:21.272662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5836fdc3-f30b-4031-b367-5060808e6225 admin admin] PATH_INFO: `/v3/projects/2f62933282d04a1286b0ea9d7890c2ba/users/33764a68f463451c99b1889239b1d9c9/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:21.288426 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5836fdc3-f30b-4031-b367-5060808e6225 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2f62933282d04a1286b0ea9d7890c2ba, user_id=33764a68f463451c99b1889239b1d9c9, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:21.295007 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5836fdc3-f30b-4031-b367-5060808e6225 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:21.357949 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1282/2567] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:21 2022] PUT /identity/v3/projects/2f62933282d04a1286b0ea9d7890c2ba/users/33764a68f463451c99b1889239b1d9c9/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:21.368491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7623fb4a-8da3-4201-b620-deb4b2028941 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:21.368773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7623fb4a-8da3-4201-b620-deb4b2028941 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:21.369044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7623fb4a-8da3-4201-b620-deb4b2028941 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:21.398711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-7623fb4a-8da3-4201-b620-deb4b2028941 None None] MFA Rules not processed for user `33764a68f463451c99b1889239b1d9c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:21.399940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-7623fb4a-8da3-4201-b620-deb4b2028941 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:21.409119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:21.463964 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1286/2568] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:21 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:21.468872 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5836fdc3-f30b-4031-b367-5060808e6225 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:21.482261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:21.532012 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5836fdc3-f30b-4031-b367-5060808e6225 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:21.537091 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8bff7a-385d-4aa3-8608-f8c3db581507 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:21.537781 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8bff7a-385d-4aa3-8608-f8c3db581507 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:21.538057 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8bff7a-385d-4aa3-8608-f8c3db581507 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:21.538321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8bff7a-385d-4aa3-8608-f8c3db581507 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:21.539006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8bff7a-385d-4aa3-8608-f8c3db581507 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:21.541019 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8bff7a-385d-4aa3-8608-f8c3db581507 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:21.672813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1283/2569] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:33:21 2022] POST /identity/v3/projects => generated 357 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:21.677648 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7623fb4a-8da3-4201-b620-deb4b2028941 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:21.690463 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7623fb4a-8da3-4201-b620-deb4b2028941 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:21.756945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e53c23c-ece3-46f7-9e7f-cba09ee38910 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:21.757554 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e53c23c-ece3-46f7-9e7f-cba09ee38910 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:21.757826 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e53c23c-ece3-46f7-9e7f-cba09ee38910 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:21.758101 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e53c23c-ece3-46f7-9e7f-cba09ee38910 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:21.758857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e53c23c-ece3-46f7-9e7f-cba09ee38910 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:21.760878 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e53c23c-ece3-46f7-9e7f-cba09ee38910 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:21.839168 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1287/2570] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:33:21 2022] POST /identity/v3/users => generated 285 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:33:21.843734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8bff7a-385d-4aa3-8608-f8c3db581507 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:21.856834 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8bff7a-385d-4aa3-8608-f8c3db581507 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:21.863754 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8243b733-5907-40f3-b257-abfd6cd59f28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:21.864279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8243b733-5907-40f3-b257-abfd6cd59f28 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:21.864556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8243b733-5907-40f3-b257-abfd6cd59f28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:21.864828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8243b733-5907-40f3-b257-abfd6cd59f28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:21.865547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8243b733-5907-40f3-b257-abfd6cd59f28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:21.867262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8243b733-5907-40f3-b257-abfd6cd59f28 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:21.879538 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1284/2571] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:21 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:21.886711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e53c23c-ece3-46f7-9e7f-cba09ee38910 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:21.898472 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e53c23c-ece3-46f7-9e7f-cba09ee38910 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:21.903110 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb4ed91-e3ce-4f47-b2ed-6779af237c27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:21.903765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb4ed91-e3ce-4f47-b2ed-6779af237c27 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:21.904024 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb4ed91-e3ce-4f47-b2ed-6779af237c27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:21.904326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb4ed91-e3ce-4f47-b2ed-6779af237c27 admin admin] PATH_INFO: `/v3/projects/cda5afc8b72d4f67a78ca542b83d82aa/users/13ae1300debf499fa7facea27ad79cfd/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:21.919761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb4ed91-e3ce-4f47-b2ed-6779af237c27 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cda5afc8b72d4f67a78ca542b83d82aa, user_id=13ae1300debf499fa7facea27ad79cfd, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:21.922469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb4ed91-e3ce-4f47-b2ed-6779af237c27 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:22.015033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1288/2572] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:21 2022] PUT /identity/v3/projects/cda5afc8b72d4f67a78ca542b83d82aa/users/13ae1300debf499fa7facea27ad79cfd/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:22.019300 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8243b733-5907-40f3-b257-abfd6cd59f28 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:22.029720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8243b733-5907-40f3-b257-abfd6cd59f28 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:22.088257 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd399436-504c-4734-8e3c-4065ce9ce788 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:22.088789 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd399436-504c-4734-8e3c-4065ce9ce788 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:22.089068 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd399436-504c-4734-8e3c-4065ce9ce788 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:22.089338 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd399436-504c-4734-8e3c-4065ce9ce788 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:22.089970 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd399436-504c-4734-8e3c-4065ce9ce788 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:22.091620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd399436-504c-4734-8e3c-4065ce9ce788 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:22.102355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1285/2573] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:22 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:22.106554 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb4ed91-e3ce-4f47-b2ed-6779af237c27 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:22.117832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb4ed91-e3ce-4f47-b2ed-6779af237c27 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:22.122582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36046dd8-d325-4520-a5cc-4cb60710ac79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:22.123220 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36046dd8-d325-4520-a5cc-4cb60710ac79 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:22.123521 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36046dd8-d325-4520-a5cc-4cb60710ac79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:22.123789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36046dd8-d325-4520-a5cc-4cb60710ac79 admin admin] PATH_INFO: `/v3/projects/cda5afc8b72d4f67a78ca542b83d82aa/users/13ae1300debf499fa7facea27ad79cfd/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:22.125835 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36046dd8-d325-4520-a5cc-4cb60710ac79 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cda5afc8b72d4f67a78ca542b83d82aa, user_id=13ae1300debf499fa7facea27ad79cfd, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:22.128604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36046dd8-d325-4520-a5cc-4cb60710ac79 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:22.139124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1289/2574] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:22 2022] PUT /identity/v3/projects/cda5afc8b72d4f67a78ca542b83d82aa/users/13ae1300debf499fa7facea27ad79cfd/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:22.149232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e72a68c-4c1d-4888-a983-c858664f1257 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:22.149571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e72a68c-4c1d-4888-a983-c858664f1257 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:22.149831 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e72a68c-4c1d-4888-a983-c858664f1257 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:22.179290 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-4e72a68c-4c1d-4888-a983-c858664f1257 None None] MFA Rules not processed for user `13ae1300debf499fa7facea27ad79cfd`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:22.180544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-4e72a68c-4c1d-4888-a983-c858664f1257 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:22.249888 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1286/2575] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:22 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:22.261210 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36046dd8-d325-4520-a5cc-4cb60710ac79 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:22.264469 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-36046dd8-d325-4520-a5cc-4cb60710ac79 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:22.324346 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36046dd8-d325-4520-a5cc-4cb60710ac79 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:22.328082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66aaf59-26af-44d3-8aa0-78df94a545d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:22.328710 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66aaf59-26af-44d3-8aa0-78df94a545d1 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:22.328986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66aaf59-26af-44d3-8aa0-78df94a545d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:22.329252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66aaf59-26af-44d3-8aa0-78df94a545d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:22.339294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d66aaf59-26af-44d3-8aa0-78df94a545d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:22.340755 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d66aaf59-26af-44d3-8aa0-78df94a545d1 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:22.351024 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1290/2576] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:22 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:22.531626 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e72a68c-4c1d-4888-a983-c858664f1257 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:22.541498 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e72a68c-4c1d-4888-a983-c858664f1257 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:22.545190 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af426f13-e10b-479d-afef-5e2d58f6a724 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:22.545786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af426f13-e10b-479d-afef-5e2d58f6a724 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:22.546063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af426f13-e10b-479d-afef-5e2d58f6a724 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:22.546332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af426f13-e10b-479d-afef-5e2d58f6a724 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:22.553334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:22.620925 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af426f13-e10b-479d-afef-5e2d58f6a724 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:22.622266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af426f13-e10b-479d-afef-5e2d58f6a724 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:22.634169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1287/2577] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:22 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:30.175101 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66aaf59-26af-44d3-8aa0-78df94a545d1 service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:30.176618 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-d66aaf59-26af-44d3-8aa0-78df94a545d1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:30.186813 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:30.247068 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66aaf59-26af-44d3-8aa0-78df94a545d1 service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:30.251735 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c00d98-7160-4f8c-a719-27ebc36e525d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:30.252404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c00d98-7160-4f8c-a719-27ebc36e525d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:30.252710 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c00d98-7160-4f8c-a719-27ebc36e525d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:30.252965 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c00d98-7160-4f8c-a719-27ebc36e525d admin admin] PATH_INFO: `/v3/users/3f6fcc97f82e481581f60f936ad3e4fa` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:30.254378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c00d98-7160-4f8c-a719-27ebc36e525d admin admin] RBAC: Authorizing `identity:delete_user(user_id=3f6fcc97f82e481581f60f936ad3e4fa)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:30.256880 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c00d98-7160-4f8c-a719-27ebc36e525d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:30.491084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-79c00d98-7160-4f8c-a719-27ebc36e525d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3f6fcc97f82e481581f60f936ad3e4fa'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:30.498007 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-79c00d98-7160-4f8c-a719-27ebc36e525d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3f6fcc97f82e481581f60f936ad3e4fa'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:30.510159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-79c00d98-7160-4f8c-a719-27ebc36e525d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3f6fcc97f82e481581f60f936ad3e4fa'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:30.602339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-79c00d98-7160-4f8c-a719-27ebc36e525d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3f6fcc97f82e481581f60f936ad3e4fa'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:30.602864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-79c00d98-7160-4f8c-a719-27ebc36e525d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3f6fcc97f82e481581f60f936ad3e4fa'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:30.609513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1291/2578] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:30 2022] DELETE /identity/v3/users/3f6fcc97f82e481581f60f936ad3e4fa => generated 0 bytes in 435 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:30.614124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf130e0-3354-40b7-87c2-dc5be1970b88 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:30.614613 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf130e0-3354-40b7-87c2-dc5be1970b88 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:30.614712 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf130e0-3354-40b7-87c2-dc5be1970b88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:30.633751 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-9bf130e0-3354-40b7-87c2-dc5be1970b88 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:30.634976 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9bf130e0-3354-40b7-87c2-dc5be1970b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:30.701586 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1288/2579] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:30.711938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c00d98-7160-4f8c-a719-27ebc36e525d admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:30.713254 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-79c00d98-7160-4f8c-a719-27ebc36e525d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:30.771137 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c00d98-7160-4f8c-a719-27ebc36e525d admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:30.774814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18d2c33-c550-48ef-ad29-b47eb9518613 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:30.775413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18d2c33-c550-48ef-ad29-b47eb9518613 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:30.775689 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18d2c33-c550-48ef-ad29-b47eb9518613 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:30.775959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18d2c33-c550-48ef-ad29-b47eb9518613 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:30.785000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18d2c33-c550-48ef-ad29-b47eb9518613 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:30.786429 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18d2c33-c550-48ef-ad29-b47eb9518613 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:30.796159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1292/2580] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:31.187591 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf130e0-3354-40b7-87c2-dc5be1970b88 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:31.189108 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9bf130e0-3354-40b7-87c2-dc5be1970b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:31.201762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf130e0-3354-40b7-87c2-dc5be1970b88 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:31.207253 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:31.207891 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:31.208205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:31.209910 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384 admin admin] PATH_INFO: `/v3/projects/909a4c4d763b464680c5beda9eaea6dc` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:31.209910 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384 admin admin] RBAC: Authorizing `identity:delete_project(project_id=909a4c4d763b464680c5beda9eaea6dc)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:31.211714 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:31.307848 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '909a4c4d763b464680c5beda9eaea6dc', 'request_id': 'req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:31.313930 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '909a4c4d763b464680c5beda9eaea6dc', 'request_id': 'req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:31.321827 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '909a4c4d763b464680c5beda9eaea6dc', 'request_id': 'req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:31.442020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384 admin admin] The token cache is being invalidate because project 909a4c4d763b464680c5beda9eaea6dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:31.442020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:31.446331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:31.446331 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1289/2581] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:31 2022] DELETE /identity/v3/projects/909a4c4d763b464680c5beda9eaea6dc => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:31.447655 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18d2c33-c550-48ef-ad29-b47eb9518613 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:31.458602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18d2c33-c550-48ef-ad29-b47eb9518613 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:31.492073 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:31.528527 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66e435d-4dca-4b32-9b1a-0ea295efbcfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:31.534580 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66e435d-4dca-4b32-9b1a-0ea295efbcfd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:31.534580 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66e435d-4dca-4b32-9b1a-0ea295efbcfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:31.534580 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66e435d-4dca-4b32-9b1a-0ea295efbcfd admin admin] PATH_INFO: `/v3/users/48f25c002e2e4a32aaf18bb7c0685ad6` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:31.534580 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66e435d-4dca-4b32-9b1a-0ea295efbcfd admin admin] RBAC: Authorizing `identity:delete_user(user_id=48f25c002e2e4a32aaf18bb7c0685ad6)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:31.534580 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66e435d-4dca-4b32-9b1a-0ea295efbcfd admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:31.800599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f66e435d-4dca-4b32-9b1a-0ea295efbcfd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '48f25c002e2e4a32aaf18bb7c0685ad6'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:31.807434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f66e435d-4dca-4b32-9b1a-0ea295efbcfd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '48f25c002e2e4a32aaf18bb7c0685ad6'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:31.820676 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f66e435d-4dca-4b32-9b1a-0ea295efbcfd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '48f25c002e2e4a32aaf18bb7c0685ad6'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:31.877358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f66e435d-4dca-4b32-9b1a-0ea295efbcfd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '48f25c002e2e4a32aaf18bb7c0685ad6'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:31.878407 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f66e435d-4dca-4b32-9b1a-0ea295efbcfd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '48f25c002e2e4a32aaf18bb7c0685ad6'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:31.885927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1293/2582] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:31 2022] DELETE /identity/v3/users/48f25c002e2e4a32aaf18bb7c0685ad6 => generated 0 bytes in 439 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:32.388000 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:32.389539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:32.402962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e53205f-d3e8-43b6-b1fb-f5bc1819a384 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:32.431521 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:32.466848 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c590d29-6477-45c4-ace9-97db1bf1021c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:32.467485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c590d29-6477-45c4-ace9-97db1bf1021c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:32.467766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c590d29-6477-45c4-ace9-97db1bf1021c admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:32.468069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c590d29-6477-45c4-ace9-97db1bf1021c admin admin] PATH_INFO: `/v3/projects/9e6609eb6e0f4fe6904798711cd81275` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:32.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c590d29-6477-45c4-ace9-97db1bf1021c admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e6609eb6e0f4fe6904798711cd81275)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:32.471132 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c590d29-6477-45c4-ace9-97db1bf1021c admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:32.579353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1c590d29-6477-45c4-ace9-97db1bf1021c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e6609eb6e0f4fe6904798711cd81275', 'request_id': 'req-1c590d29-6477-45c4-ace9-97db1bf1021c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:32.585270 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1c590d29-6477-45c4-ace9-97db1bf1021c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e6609eb6e0f4fe6904798711cd81275', 'request_id': 'req-1c590d29-6477-45c4-ace9-97db1bf1021c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:32.591529 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1c590d29-6477-45c4-ace9-97db1bf1021c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e6609eb6e0f4fe6904798711cd81275', 'request_id': 'req-1c590d29-6477-45c4-ace9-97db1bf1021c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:32.671179 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1c590d29-6477-45c4-ace9-97db1bf1021c admin admin] The token cache is being invalidate because project 9e6609eb6e0f4fe6904798711cd81275 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:32.671388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1c590d29-6477-45c4-ace9-97db1bf1021c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:32.672038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1c590d29-6477-45c4-ace9-97db1bf1021c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:32.673600 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1290/2583] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:32 2022] DELETE /identity/v3/projects/9e6609eb6e0f4fe6904798711cd81275 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:32.679049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcd0f99-4a69-4a4b-9b21-00ffb4ac2416 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:32.679516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcd0f99-4a69-4a4b-9b21-00ffb4ac2416 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:32.679614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcd0f99-4a69-4a4b-9b21-00ffb4ac2416 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:32.700555 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-fdcd0f99-4a69-4a4b-9b21-00ffb4ac2416 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:32.701811 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-fdcd0f99-4a69-4a4b-9b21-00ffb4ac2416 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:32.763001 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1294/2584] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:32.770862 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ef0e27-1ef8-47cb-9183-b3983ed7d3e6 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:32.771141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ef0e27-1ef8-47cb-9183-b3983ed7d3e6 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:32.771412 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ef0e27-1ef8-47cb-9183-b3983ed7d3e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:32.791255 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-a9ef0e27-1ef8-47cb-9183-b3983ed7d3e6 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:32.792513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-a9ef0e27-1ef8-47cb-9183-b3983ed7d3e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:32.800495 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1291/2585] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:32.805324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcd0f99-4a69-4a4b-9b21-00ffb4ac2416 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:32.814996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcd0f99-4a69-4a4b-9b21-00ffb4ac2416 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:32.820186 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a271a9b3-9940-4214-9fd0-41a81a5ab352 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:32.824919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a271a9b3-9940-4214-9fd0-41a81a5ab352 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:32.824919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a271a9b3-9940-4214-9fd0-41a81a5ab352 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:32.824919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a271a9b3-9940-4214-9fd0-41a81a5ab352 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:32.824919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a271a9b3-9940-4214-9fd0-41a81a5ab352 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:33:32.824919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a271a9b3-9940-4214-9fd0-41a81a5ab352 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:32.824919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a271a9b3-9940-4214-9fd0-41a81a5ab352 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:32.836622 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1295/2586] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:33:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:33:32.841321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ef0e27-1ef8-47cb-9183-b3983ed7d3e6 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:32.851810 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ef0e27-1ef8-47cb-9183-b3983ed7d3e6 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:32.856931 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af7aed2-9921-4218-bde6-b94c9caadc94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:32.857677 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af7aed2-9921-4218-bde6-b94c9caadc94 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:32.857955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af7aed2-9921-4218-bde6-b94c9caadc94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:32.858227 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af7aed2-9921-4218-bde6-b94c9caadc94 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:32.858870 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af7aed2-9921-4218-bde6-b94c9caadc94 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:32.864078 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af7aed2-9921-4218-bde6-b94c9caadc94 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:32.971215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1292/2587] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:33:32 2022] POST /identity/v3/projects => generated 385 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:32.976294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a271a9b3-9940-4214-9fd0-41a81a5ab352 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:32.987002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a271a9b3-9940-4214-9fd0-41a81a5ab352 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:33.019279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:33.044162 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074b1b6d-c370-401b-ad97-6222aa62d04f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:33.044842 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074b1b6d-c370-401b-ad97-6222aa62d04f admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:33.045126 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074b1b6d-c370-401b-ad97-6222aa62d04f admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:33.045399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074b1b6d-c370-401b-ad97-6222aa62d04f admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:33.046018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074b1b6d-c370-401b-ad97-6222aa62d04f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:33.047929 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074b1b6d-c370-401b-ad97-6222aa62d04f admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:33.185173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1296/2588] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:33:32 2022] POST /identity/v3/users => generated 293 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:33.189575 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af7aed2-9921-4218-bde6-b94c9caadc94 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:33.200835 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af7aed2-9921-4218-bde6-b94c9caadc94 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:33.205439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b156e9-f8f2-469a-bac4-f2c81e1b60d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:33.206055 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b156e9-f8f2-469a-bac4-f2c81e1b60d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:33.206318 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b156e9-f8f2-469a-bac4-f2c81e1b60d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:33.206838 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b156e9-f8f2-469a-bac4-f2c81e1b60d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:33.207733 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b156e9-f8f2-469a-bac4-f2c81e1b60d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:33.214713 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b156e9-f8f2-469a-bac4-f2c81e1b60d5 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:33.226092 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1293/2589] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:33 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:33.226769 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074b1b6d-c370-401b-ad97-6222aa62d04f admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:33.242034 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074b1b6d-c370-401b-ad97-6222aa62d04f admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:33.246941 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b2dd37-0272-4618-822e-5519e4403a9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:33.247780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b2dd37-0272-4618-822e-5519e4403a9f admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:33.248054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b2dd37-0272-4618-822e-5519e4403a9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:33.249079 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b2dd37-0272-4618-822e-5519e4403a9f admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:33.249638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b2dd37-0272-4618-822e-5519e4403a9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:33.251637 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b2dd37-0272-4618-822e-5519e4403a9f admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:33.263397 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1297/2590] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:33 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:33:33.267342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b156e9-f8f2-469a-bac4-f2c81e1b60d5 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:33.277662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b156e9-f8f2-469a-bac4-f2c81e1b60d5 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:33.282202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a867b6-ecea-49e6-9644-b8c8fefc8c69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:33.282854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a867b6-ecea-49e6-9644-b8c8fefc8c69 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:33.283128 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a867b6-ecea-49e6-9644-b8c8fefc8c69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:33.283395 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a867b6-ecea-49e6-9644-b8c8fefc8c69 admin admin] PATH_INFO: `/v3/projects/6aacdd49d4ae4f9e81e82188a441a9ca/users/c10054d60b7a42c9ba62988ba3888b3b/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:33.305196 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a867b6-ecea-49e6-9644-b8c8fefc8c69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6aacdd49d4ae4f9e81e82188a441a9ca, user_id=c10054d60b7a42c9ba62988ba3888b3b, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:33.307899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a867b6-ecea-49e6-9644-b8c8fefc8c69 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:33.413540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1294/2591] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:33 2022] PUT /identity/v3/projects/6aacdd49d4ae4f9e81e82188a441a9ca/users/c10054d60b7a42c9ba62988ba3888b3b/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:33.424857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689cc83b-76cb-4835-b5f5-fc84f294bd76 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:33.425141 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689cc83b-76cb-4835-b5f5-fc84f294bd76 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:33.425414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689cc83b-76cb-4835-b5f5-fc84f294bd76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:33.458726 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-689cc83b-76cb-4835-b5f5-fc84f294bd76 None None] MFA Rules not processed for user `c10054d60b7a42c9ba62988ba3888b3b`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:33.460521 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-689cc83b-76cb-4835-b5f5-fc84f294bd76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:33.531378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1298/2592] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:33 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:33.537378 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a867b6-ecea-49e6-9644-b8c8fefc8c69 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:33.601851 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:33.608182 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a867b6-ecea-49e6-9644-b8c8fefc8c69 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:33.613697 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3b6e21-157a-43af-bd58-1bc5387b9fca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:33.614200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3b6e21-157a-43af-bd58-1bc5387b9fca admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:33.614534 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3b6e21-157a-43af-bd58-1bc5387b9fca admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:33.614821 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3b6e21-157a-43af-bd58-1bc5387b9fca admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:33.616319 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3b6e21-157a-43af-bd58-1bc5387b9fca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:33.618544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3b6e21-157a-43af-bd58-1bc5387b9fca admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:33.705290 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1295/2593] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:33:33 2022] POST /identity/v3/projects => generated 385 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:33.707398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689cc83b-76cb-4835-b5f5-fc84f294bd76 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:33.718131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689cc83b-76cb-4835-b5f5-fc84f294bd76 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:33.775125 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cf6930-075f-4f3d-9845-dbf126fe7604 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:33.775645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cf6930-075f-4f3d-9845-dbf126fe7604 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:33.775919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cf6930-075f-4f3d-9845-dbf126fe7604 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:33.776316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cf6930-075f-4f3d-9845-dbf126fe7604 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:33.776860 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cf6930-075f-4f3d-9845-dbf126fe7604 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:33.778744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cf6930-075f-4f3d-9845-dbf126fe7604 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:33.881486 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1299/2594] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:33:33 2022] POST /identity/v3/users => generated 299 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:33.882890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3b6e21-157a-43af-bd58-1bc5387b9fca admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:33.893779 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3b6e21-157a-43af-bd58-1bc5387b9fca admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:33.898426 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebbcfe7-14c2-4127-8c74-2cfe96fd79d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:33.899029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebbcfe7-14c2-4127-8c74-2cfe96fd79d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:33.899304 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebbcfe7-14c2-4127-8c74-2cfe96fd79d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:33.899570 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebbcfe7-14c2-4127-8c74-2cfe96fd79d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:33.900362 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebbcfe7-14c2-4127-8c74-2cfe96fd79d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:33.901939 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebbcfe7-14c2-4127-8c74-2cfe96fd79d1 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:33.919859 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1296/2595] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:33 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:33:33.921484 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cf6930-075f-4f3d-9845-dbf126fe7604 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:33.934166 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cf6930-075f-4f3d-9845-dbf126fe7604 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:33.939364 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db90d7c-f96f-4b67-aa44-b944b9082576 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:33.940212 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db90d7c-f96f-4b67-aa44-b944b9082576 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:33.940662 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db90d7c-f96f-4b67-aa44-b944b9082576 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:33.941098 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db90d7c-f96f-4b67-aa44-b944b9082576 admin admin] PATH_INFO: `/v3/projects/52ed86df8db947a7aeed9ba65a6be1a1/users/e9b741bfa2114da8bbadc7aa7dbe0ce5/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:33.960827 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db90d7c-f96f-4b67-aa44-b944b9082576 admin admin] RBAC: Authorizing `identity:create_grant(project_id=52ed86df8db947a7aeed9ba65a6be1a1, user_id=e9b741bfa2114da8bbadc7aa7dbe0ce5, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:33.964337 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db90d7c-f96f-4b67-aa44-b944b9082576 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:34.049855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1300/2596] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:33 2022] PUT /identity/v3/projects/52ed86df8db947a7aeed9ba65a6be1a1/users/e9b741bfa2114da8bbadc7aa7dbe0ce5/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:33:34.054119 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebbcfe7-14c2-4127-8c74-2cfe96fd79d1 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:34.064546 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebbcfe7-14c2-4127-8c74-2cfe96fd79d1 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:34.125544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a87e61d-5c70-40fb-9962-99336cd52b13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:34.126154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a87e61d-5c70-40fb-9962-99336cd52b13 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:34.126438 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a87e61d-5c70-40fb-9962-99336cd52b13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:34.126772 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a87e61d-5c70-40fb-9962-99336cd52b13 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:34.127367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a87e61d-5c70-40fb-9962-99336cd52b13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:34.129533 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a87e61d-5c70-40fb-9962-99336cd52b13 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:34.139262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1297/2597] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:34 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:34.143311 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db90d7c-f96f-4b67-aa44-b944b9082576 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:34.153741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db90d7c-f96f-4b67-aa44-b944b9082576 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:34.158731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6817b7bc-1f68-4c3c-ace0-e58328e8865e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:34.159432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6817b7bc-1f68-4c3c-ace0-e58328e8865e admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:34.159725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6817b7bc-1f68-4c3c-ace0-e58328e8865e admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:34.159986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6817b7bc-1f68-4c3c-ace0-e58328e8865e admin admin] PATH_INFO: `/v3/projects/52ed86df8db947a7aeed9ba65a6be1a1/users/e9b741bfa2114da8bbadc7aa7dbe0ce5/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:34.162462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6817b7bc-1f68-4c3c-ace0-e58328e8865e admin admin] RBAC: Authorizing `identity:create_grant(project_id=52ed86df8db947a7aeed9ba65a6be1a1, user_id=e9b741bfa2114da8bbadc7aa7dbe0ce5, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:34.165234 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6817b7bc-1f68-4c3c-ace0-e58328e8865e admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:34.168359 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:34.175819 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1301/2598] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:34 2022] PUT /identity/v3/projects/52ed86df8db947a7aeed9ba65a6be1a1/users/e9b741bfa2114da8bbadc7aa7dbe0ce5/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:34.186668 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31db1c48-9cce-4886-84bc-d9bde6876dfb None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:34.187040 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31db1c48-9cce-4886-84bc-d9bde6876dfb None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:34.187310 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31db1c48-9cce-4886-84bc-d9bde6876dfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:34.215843 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-31db1c48-9cce-4886-84bc-d9bde6876dfb None None] MFA Rules not processed for user `e9b741bfa2114da8bbadc7aa7dbe0ce5`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:34.217093 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-31db1c48-9cce-4886-84bc-d9bde6876dfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:34.284983 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1298/2599] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:34 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:34.297006 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6817b7bc-1f68-4c3c-ace0-e58328e8865e admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:34.298690 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6817b7bc-1f68-4c3c-ace0-e58328e8865e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:34.357458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6817b7bc-1f68-4c3c-ace0-e58328e8865e admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:34.361090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e766875-0e7c-40e9-9e7e-7827610aae72 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:34.361687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e766875-0e7c-40e9-9e7e-7827610aae72 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:34.361963 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e766875-0e7c-40e9-9e7e-7827610aae72 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:34.362232 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e766875-0e7c-40e9-9e7e-7827610aae72 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:34.371644 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e766875-0e7c-40e9-9e7e-7827610aae72 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:34.373084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e766875-0e7c-40e9-9e7e-7827610aae72 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:34.382381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1302/2600] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:34 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:35.459419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31db1c48-9cce-4886-84bc-d9bde6876dfb None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:35.517071 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:35.530807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31db1c48-9cce-4886-84bc-d9bde6876dfb None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:35.535352 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb5cda6-910d-47f1-ab11-f0185bd17b0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:35.536051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb5cda6-910d-47f1-ab11-f0185bd17b0f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:35.536346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb5cda6-910d-47f1-ab11-f0185bd17b0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:35.536616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb5cda6-910d-47f1-ab11-f0185bd17b0f admin admin] PATH_INFO: `/v3/users/c10054d60b7a42c9ba62988ba3888b3b` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:35.538259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb5cda6-910d-47f1-ab11-f0185bd17b0f admin admin] RBAC: Authorizing `identity:delete_user(user_id=c10054d60b7a42c9ba62988ba3888b3b)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:35.540213 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb5cda6-910d-47f1-ab11-f0185bd17b0f admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:35.742086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7fb5cda6-910d-47f1-ab11-f0185bd17b0f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c10054d60b7a42c9ba62988ba3888b3b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:35.749121 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7fb5cda6-910d-47f1-ab11-f0185bd17b0f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c10054d60b7a42c9ba62988ba3888b3b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:35.749818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7fb5cda6-910d-47f1-ab11-f0185bd17b0f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c10054d60b7a42c9ba62988ba3888b3b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:35.762032 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7fb5cda6-910d-47f1-ab11-f0185bd17b0f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c10054d60b7a42c9ba62988ba3888b3b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:35.843097 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7fb5cda6-910d-47f1-ab11-f0185bd17b0f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c10054d60b7a42c9ba62988ba3888b3b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:35.850424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1299/2601] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:35 2022] DELETE /identity/v3/users/c10054d60b7a42c9ba62988ba3888b3b => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:35.855232 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be694cd-3638-474d-ae2d-07cb0395df67 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:35.855232 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be694cd-3638-474d-ae2d-07cb0395df67 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:35.855802 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be694cd-3638-474d-ae2d-07cb0395df67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:35.876179 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-6be694cd-3638-474d-ae2d-07cb0395df67 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:35.877492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6be694cd-3638-474d-ae2d-07cb0395df67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:35.940063 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1303/2602] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:33:35.950295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb5cda6-910d-47f1-ab11-f0185bd17b0f admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:35.955315 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7fb5cda6-910d-47f1-ab11-f0185bd17b0f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:36.017594 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb5cda6-910d-47f1-ab11-f0185bd17b0f admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:36.021878 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d67a849-11a8-4cbf-a996-7a62b0bbe303 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:36.022548 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d67a849-11a8-4cbf-a996-7a62b0bbe303 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:36.022824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d67a849-11a8-4cbf-a996-7a62b0bbe303 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:36.023782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d67a849-11a8-4cbf-a996-7a62b0bbe303 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:36.033241 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d67a849-11a8-4cbf-a996-7a62b0bbe303 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:36.034666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d67a849-11a8-4cbf-a996-7a62b0bbe303 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:36.044954 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1300/2603] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:36.534871 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be694cd-3638-474d-ae2d-07cb0395df67 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:36.536383 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6be694cd-3638-474d-ae2d-07cb0395df67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:36.555231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be694cd-3638-474d-ae2d-07cb0395df67 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:36.561566 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b72a68-8701-4811-9217-639819dbf48d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:36.562223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b72a68-8701-4811-9217-639819dbf48d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:36.563029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b72a68-8701-4811-9217-639819dbf48d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:36.563029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b72a68-8701-4811-9217-639819dbf48d admin admin] PATH_INFO: `/v3/projects/6aacdd49d4ae4f9e81e82188a441a9ca` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:36.567615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b72a68-8701-4811-9217-639819dbf48d admin admin] RBAC: Authorizing `identity:delete_project(project_id=6aacdd49d4ae4f9e81e82188a441a9ca)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:36.570552 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b72a68-8701-4811-9217-639819dbf48d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:36.705708 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b2b72a68-8701-4811-9217-639819dbf48d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6aacdd49d4ae4f9e81e82188a441a9ca', 'request_id': 'req-b2b72a68-8701-4811-9217-639819dbf48d'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:36.820060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b2b72a68-8701-4811-9217-639819dbf48d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6aacdd49d4ae4f9e81e82188a441a9ca', 'request_id': 'req-b2b72a68-8701-4811-9217-639819dbf48d'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:36.829700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b2b72a68-8701-4811-9217-639819dbf48d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6aacdd49d4ae4f9e81e82188a441a9ca', 'request_id': 'req-b2b72a68-8701-4811-9217-639819dbf48d'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:36.842849 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b2b72a68-8701-4811-9217-639819dbf48d admin admin] The token cache is being invalidate because project 6aacdd49d4ae4f9e81e82188a441a9ca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:36.843388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b2b72a68-8701-4811-9217-639819dbf48d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:36.844248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-b2b72a68-8701-4811-9217-639819dbf48d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:36.845887 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1304/2604] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:36 2022] DELETE /identity/v3/projects/6aacdd49d4ae4f9e81e82188a441a9ca => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:36.850084 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d67a849-11a8-4cbf-a996-7a62b0bbe303 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:36.862974 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d67a849-11a8-4cbf-a996-7a62b0bbe303 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:36.939233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6e16d9-2fae-46be-b6e6-c6b11a70ec7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:36.939791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6e16d9-2fae-46be-b6e6-c6b11a70ec7a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:36.940072 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6e16d9-2fae-46be-b6e6-c6b11a70ec7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:36.940390 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6e16d9-2fae-46be-b6e6-c6b11a70ec7a admin admin] PATH_INFO: `/v3/users/e9b741bfa2114da8bbadc7aa7dbe0ce5` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:36.941571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6e16d9-2fae-46be-b6e6-c6b11a70ec7a admin admin] RBAC: Authorizing `identity:delete_user(user_id=e9b741bfa2114da8bbadc7aa7dbe0ce5)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:36.944314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6e16d9-2fae-46be-b6e6-c6b11a70ec7a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:36.959731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:37.140597 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ec6e16d9-2fae-46be-b6e6-c6b11a70ec7a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e9b741bfa2114da8bbadc7aa7dbe0ce5'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:37.147674 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ec6e16d9-2fae-46be-b6e6-c6b11a70ec7a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e9b741bfa2114da8bbadc7aa7dbe0ce5'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:37.148518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ec6e16d9-2fae-46be-b6e6-c6b11a70ec7a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e9b741bfa2114da8bbadc7aa7dbe0ce5'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:37.161061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ec6e16d9-2fae-46be-b6e6-c6b11a70ec7a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e9b741bfa2114da8bbadc7aa7dbe0ce5'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:37.240953 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b72a68-8701-4811-9217-639819dbf48d admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:37.242396 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-b2b72a68-8701-4811-9217-639819dbf48d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:37.258852 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ec6e16d9-2fae-46be-b6e6-c6b11a70ec7a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e9b741bfa2114da8bbadc7aa7dbe0ce5'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:37.269661 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1301/2605] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:36 2022] DELETE /identity/v3/users/e9b741bfa2114da8bbadc7aa7dbe0ce5 => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:37.279160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:37.368670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b72a68-8701-4811-9217-639819dbf48d admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:37.434454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77677599-019d-4980-b222-273fe501b254 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '972b0f1dbfef4d85a6dfc4c5fd839bde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:37.435069 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77677599-019d-4980-b222-273fe501b254 service placement] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:37.435344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77677599-019d-4980-b222-273fe501b254 service placement] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:37.435615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77677599-019d-4980-b222-273fe501b254 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:37.437499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-77677599-019d-4980-b222-273fe501b254 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:37.449154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77677599-019d-4980-b222-273fe501b254 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:37.450816 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77677599-019d-4980-b222-273fe501b254 service placement] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:37.468988 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1305/2606] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:37 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:33:37.734490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6e16d9-2fae-46be-b6e6-c6b11a70ec7a admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:37.734490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ec6e16d9-2fae-46be-b6e6-c6b11a70ec7a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:37.807469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6e16d9-2fae-46be-b6e6-c6b11a70ec7a admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:37.812217 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728a4c42-7b7e-46f6-bcf6-67a572095333 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:37.812964 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728a4c42-7b7e-46f6-bcf6-67a572095333 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:37.813287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728a4c42-7b7e-46f6-bcf6-67a572095333 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:37.813617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728a4c42-7b7e-46f6-bcf6-67a572095333 admin admin] PATH_INFO: `/v3/projects/52ed86df8db947a7aeed9ba65a6be1a1` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:37.814738 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728a4c42-7b7e-46f6-bcf6-67a572095333 admin admin] RBAC: Authorizing `identity:delete_project(project_id=52ed86df8db947a7aeed9ba65a6be1a1)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:37.816689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728a4c42-7b7e-46f6-bcf6-67a572095333 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:37.918889 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-728a4c42-7b7e-46f6-bcf6-67a572095333 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52ed86df8db947a7aeed9ba65a6be1a1', 'request_id': 'req-728a4c42-7b7e-46f6-bcf6-67a572095333'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:37.925720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-728a4c42-7b7e-46f6-bcf6-67a572095333 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52ed86df8db947a7aeed9ba65a6be1a1', 'request_id': 'req-728a4c42-7b7e-46f6-bcf6-67a572095333'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:37.932662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-728a4c42-7b7e-46f6-bcf6-67a572095333 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52ed86df8db947a7aeed9ba65a6be1a1', 'request_id': 'req-728a4c42-7b7e-46f6-bcf6-67a572095333'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:37.998437 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-728a4c42-7b7e-46f6-bcf6-67a572095333 admin admin] The token cache is being invalidate because project 52ed86df8db947a7aeed9ba65a6be1a1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:37.998854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-728a4c42-7b7e-46f6-bcf6-67a572095333 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:37.999670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-728a4c42-7b7e-46f6-bcf6-67a572095333 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:38.001794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1302/2607] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:37 2022] DELETE /identity/v3/projects/52ed86df8db947a7aeed9ba65a6be1a1 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:42.435392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77677599-019d-4980-b222-273fe501b254 service placement] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:42.436882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-77677599-019d-4980-b222-273fe501b254 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:42.450139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77677599-019d-4980-b222-273fe501b254 service placement] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:42.506876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da51f06-6fdd-4c00-8d9a-996337d7c9eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:42.507438 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da51f06-6fdd-4c00-8d9a-996337d7c9eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:42.507720 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da51f06-6fdd-4c00-8d9a-996337d7c9eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:42.507988 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da51f06-6fdd-4c00-8d9a-996337d7c9eb admin admin] PATH_INFO: `/v3/users/33764a68f463451c99b1889239b1d9c9` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:42.509175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da51f06-6fdd-4c00-8d9a-996337d7c9eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=33764a68f463451c99b1889239b1d9c9)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:42.511099 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da51f06-6fdd-4c00-8d9a-996337d7c9eb admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:42.672011 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:42.683235 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0da51f06-6fdd-4c00-8d9a-996337d7c9eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '33764a68f463451c99b1889239b1d9c9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:42.691520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0da51f06-6fdd-4c00-8d9a-996337d7c9eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '33764a68f463451c99b1889239b1d9c9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:42.705523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0da51f06-6fdd-4c00-8d9a-996337d7c9eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '33764a68f463451c99b1889239b1d9c9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:42.797683 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0da51f06-6fdd-4c00-8d9a-996337d7c9eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '33764a68f463451c99b1889239b1d9c9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:42.798409 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-0da51f06-6fdd-4c00-8d9a-996337d7c9eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '33764a68f463451c99b1889239b1d9c9'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:42.805799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1306/2608] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:42 2022] DELETE /identity/v3/users/33764a68f463451c99b1889239b1d9c9 => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:42.811084 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7249d1f4-9737-4e21-b5db-a5f718922f1f None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:42.821637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7249d1f4-9737-4e21-b5db-a5f718922f1f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:42.821637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7249d1f4-9737-4e21-b5db-a5f718922f1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:42.833162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-7249d1f4-9737-4e21-b5db-a5f718922f1f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:42.834426 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7249d1f4-9737-4e21-b5db-a5f718922f1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:42.852672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:42.897405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1303/2609] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:42.908162 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da51f06-6fdd-4c00-8d9a-996337d7c9eb admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:42.909656 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-0da51f06-6fdd-4c00-8d9a-996337d7c9eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:42.969435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da51f06-6fdd-4c00-8d9a-996337d7c9eb admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:42.973192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac8eb14-6291-4873-963f-71f1112a5bb7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:42.973825 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac8eb14-6291-4873-963f-71f1112a5bb7 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:42.974103 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac8eb14-6291-4873-963f-71f1112a5bb7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:42.974373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac8eb14-6291-4873-963f-71f1112a5bb7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:42.983753 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac8eb14-6291-4873-963f-71f1112a5bb7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:42.985221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac8eb14-6291-4873-963f-71f1112a5bb7 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:42.994936 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1307/2610] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:43.381769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7249d1f4-9737-4e21-b5db-a5f718922f1f None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:43.383759 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7249d1f4-9737-4e21-b5db-a5f718922f1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:43.395787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7249d1f4-9737-4e21-b5db-a5f718922f1f None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:43.400477 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eb7c5c-0add-454a-acc3-84725106d73c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:43.401104 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eb7c5c-0add-454a-acc3-84725106d73c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:43.401404 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eb7c5c-0add-454a-acc3-84725106d73c admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:43.401685 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eb7c5c-0add-454a-acc3-84725106d73c admin admin] PATH_INFO: `/v3/projects/2f62933282d04a1286b0ea9d7890c2ba` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:43.402821 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86eb7c5c-0add-454a-acc3-84725106d73c admin admin] RBAC: Authorizing `identity:delete_project(project_id=2f62933282d04a1286b0ea9d7890c2ba)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:43.404814 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86eb7c5c-0add-454a-acc3-84725106d73c admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:43.488892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-86eb7c5c-0add-454a-acc3-84725106d73c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2f62933282d04a1286b0ea9d7890c2ba', 'request_id': 'req-86eb7c5c-0add-454a-acc3-84725106d73c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:43.494972 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-86eb7c5c-0add-454a-acc3-84725106d73c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2f62933282d04a1286b0ea9d7890c2ba', 'request_id': 'req-86eb7c5c-0add-454a-acc3-84725106d73c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:43.501072 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-86eb7c5c-0add-454a-acc3-84725106d73c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2f62933282d04a1286b0ea9d7890c2ba', 'request_id': 'req-86eb7c5c-0add-454a-acc3-84725106d73c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:43.562428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-86eb7c5c-0add-454a-acc3-84725106d73c admin admin] The token cache is being invalidate because project 2f62933282d04a1286b0ea9d7890c2ba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:43.562705 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-86eb7c5c-0add-454a-acc3-84725106d73c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:43.563338 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-86eb7c5c-0add-454a-acc3-84725106d73c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:43.564855 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1304/2611] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:43 2022] DELETE /identity/v3/projects/2f62933282d04a1286b0ea9d7890c2ba => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:43.568925 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac8eb14-6291-4873-963f-71f1112a5bb7 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:43.579556 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac8eb14-6291-4873-963f-71f1112a5bb7 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:43.638805 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9e949f-966c-48a8-9988-8f51d847f5c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:43.639629 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9e949f-966c-48a8-9988-8f51d847f5c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:43.639629 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9e949f-966c-48a8-9988-8f51d847f5c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:43.639867 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9e949f-966c-48a8-9988-8f51d847f5c8 admin admin] PATH_INFO: `/v3/users/13ae1300debf499fa7facea27ad79cfd` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:43.641065 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9e949f-966c-48a8-9988-8f51d847f5c8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=13ae1300debf499fa7facea27ad79cfd)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:43.643007 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9e949f-966c-48a8-9988-8f51d847f5c8 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:43.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-bc9e949f-966c-48a8-9988-8f51d847f5c8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '13ae1300debf499fa7facea27ad79cfd'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:43.891170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-bc9e949f-966c-48a8-9988-8f51d847f5c8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '13ae1300debf499fa7facea27ad79cfd'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:43.894516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:43.905663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-bc9e949f-966c-48a8-9988-8f51d847f5c8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '13ae1300debf499fa7facea27ad79cfd'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:43.980534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-bc9e949f-966c-48a8-9988-8f51d847f5c8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '13ae1300debf499fa7facea27ad79cfd'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:43.981216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-bc9e949f-966c-48a8-9988-8f51d847f5c8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '13ae1300debf499fa7facea27ad79cfd'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:43.989747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1308/2612] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:43 2022] DELETE /identity/v3/users/13ae1300debf499fa7facea27ad79cfd => generated 0 bytes in 421 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:44.483002 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eb7c5c-0add-454a-acc3-84725106d73c admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:44.484510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-86eb7c5c-0add-454a-acc3-84725106d73c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:44.498020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eb7c5c-0add-454a-acc3-84725106d73c admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:44.555176 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3be152-c064-48b2-8bd5-26b147eecacf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:44.555806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3be152-c064-48b2-8bd5-26b147eecacf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:44.556080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3be152-c064-48b2-8bd5-26b147eecacf admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:44.556390 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3be152-c064-48b2-8bd5-26b147eecacf admin admin] PATH_INFO: `/v3/projects/cda5afc8b72d4f67a78ca542b83d82aa` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:44.557517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3be152-c064-48b2-8bd5-26b147eecacf admin admin] RBAC: Authorizing `identity:delete_project(project_id=cda5afc8b72d4f67a78ca542b83d82aa)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:44.559402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3be152-c064-48b2-8bd5-26b147eecacf admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:44.659116 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:44.660490 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-df3be152-c064-48b2-8bd5-26b147eecacf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cda5afc8b72d4f67a78ca542b83d82aa', 'request_id': 'req-df3be152-c064-48b2-8bd5-26b147eecacf'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:44.667358 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-df3be152-c064-48b2-8bd5-26b147eecacf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cda5afc8b72d4f67a78ca542b83d82aa', 'request_id': 'req-df3be152-c064-48b2-8bd5-26b147eecacf'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:44.677108 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-df3be152-c064-48b2-8bd5-26b147eecacf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cda5afc8b72d4f67a78ca542b83d82aa', 'request_id': 'req-df3be152-c064-48b2-8bd5-26b147eecacf'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:44.780472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-df3be152-c064-48b2-8bd5-26b147eecacf admin admin] The token cache is being invalidate because project cda5afc8b72d4f67a78ca542b83d82aa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:44.780763 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-df3be152-c064-48b2-8bd5-26b147eecacf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:44.781406 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-df3be152-c064-48b2-8bd5-26b147eecacf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:44.782917 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1305/2613] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:44 2022] DELETE /identity/v3/projects/cda5afc8b72d4f67a78ca542b83d82aa => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:44.787943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f45c61-4cbb-40e0-a100-c7a90b530982 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:44.788437 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f45c61-4cbb-40e0-a100-c7a90b530982 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:44.788720 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f45c61-4cbb-40e0-a100-c7a90b530982 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:44.809047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-29f45c61-4cbb-40e0-a100-c7a90b530982 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:44.812207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-29f45c61-4cbb-40e0-a100-c7a90b530982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:44.872767 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1309/2614] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:33:44.880860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98dfd1df-6cda-414d-8991-1734db698dd5 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:44.881141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98dfd1df-6cda-414d-8991-1734db698dd5 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:44.881417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98dfd1df-6cda-414d-8991-1734db698dd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:44.902391 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-98dfd1df-6cda-414d-8991-1734db698dd5 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:44.903635 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-98dfd1df-6cda-414d-8991-1734db698dd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:44.911727 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1306/2615] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:44.916116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f45c61-4cbb-40e0-a100-c7a90b530982 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:44.925981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f45c61-4cbb-40e0-a100-c7a90b530982 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:44.931711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc377a2c-4961-4e83-8306-87dd40a8c6f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:44.932338 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc377a2c-4961-4e83-8306-87dd40a8c6f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:44.932619 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc377a2c-4961-4e83-8306-87dd40a8c6f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:44.932894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc377a2c-4961-4e83-8306-87dd40a8c6f8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:44.933542 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc377a2c-4961-4e83-8306-87dd40a8c6f8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:33:44.933844 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc377a2c-4961-4e83-8306-87dd40a8c6f8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:44.935809 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc377a2c-4961-4e83-8306-87dd40a8c6f8 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:44.951374 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1310/2616] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:33:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:33:44.952596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98dfd1df-6cda-414d-8991-1734db698dd5 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:44.962508 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98dfd1df-6cda-414d-8991-1734db698dd5 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:44.967335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d63568-a2ce-40fa-bfd6-18064ce900b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:44.968052 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d63568-a2ce-40fa-bfd6-18064ce900b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:44.968425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d63568-a2ce-40fa-bfd6-18064ce900b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:44.968803 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d63568-a2ce-40fa-bfd6-18064ce900b8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:44.969493 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d63568-a2ce-40fa-bfd6-18064ce900b8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:44.971440 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d63568-a2ce-40fa-bfd6-18064ce900b8 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:45.051904 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1307/2617] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:33:44 2022] POST /identity/v3/projects => generated 383 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:45.057591 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc377a2c-4961-4e83-8306-87dd40a8c6f8 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:45.068794 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc377a2c-4961-4e83-8306-87dd40a8c6f8 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:45.135165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f60042-1ecf-46ba-9cc7-f542a068de4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:45.135786 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f60042-1ecf-46ba-9cc7-f542a068de4d admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:45.136072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f60042-1ecf-46ba-9cc7-f542a068de4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:45.136378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f60042-1ecf-46ba-9cc7-f542a068de4d admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:45.137021 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f60042-1ecf-46ba-9cc7-f542a068de4d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:45.139023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f60042-1ecf-46ba-9cc7-f542a068de4d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:45.224044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1311/2618] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:33:45 2022] POST /identity/v3/users => generated 292 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:45.225234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d63568-a2ce-40fa-bfd6-18064ce900b8 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:45.237686 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d63568-a2ce-40fa-bfd6-18064ce900b8 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:45.243061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8212cfc8-04b3-481c-ba75-563365644acd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:45.243688 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8212cfc8-04b3-481c-ba75-563365644acd admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:45.244074 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8212cfc8-04b3-481c-ba75-563365644acd admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:45.244256 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8212cfc8-04b3-481c-ba75-563365644acd admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:45.244894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8212cfc8-04b3-481c-ba75-563365644acd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:45.246544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8212cfc8-04b3-481c-ba75-563365644acd admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:45.259794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1308/2619] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:45.261069 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f60042-1ecf-46ba-9cc7-f542a068de4d admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:45.271240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f60042-1ecf-46ba-9cc7-f542a068de4d admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:45.276044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e9d9eb-d23b-43b5-af91-1df9cec7a7ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:45.276678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e9d9eb-d23b-43b5-af91-1df9cec7a7ef admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:45.276983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e9d9eb-d23b-43b5-af91-1df9cec7a7ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:45.277232 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e9d9eb-d23b-43b5-af91-1df9cec7a7ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:45.277882 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e9d9eb-d23b-43b5-af91-1df9cec7a7ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:45.279498 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e9d9eb-d23b-43b5-af91-1df9cec7a7ef admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:45.297997 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1312/2620] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:45 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:45.300259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8212cfc8-04b3-481c-ba75-563365644acd admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:45.311921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8212cfc8-04b3-481c-ba75-563365644acd admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:45.316925 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb16df0f-e9cc-46fc-b004-9c34ea00c99f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:45.317725 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb16df0f-e9cc-46fc-b004-9c34ea00c99f admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:45.318073 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb16df0f-e9cc-46fc-b004-9c34ea00c99f admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:45.318353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb16df0f-e9cc-46fc-b004-9c34ea00c99f admin admin] PATH_INFO: `/v3/projects/412274053ff24bdbbf7789817c7b575e/users/26969f9c613940a4b61f3afd6504eac7/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:45.334112 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb16df0f-e9cc-46fc-b004-9c34ea00c99f admin admin] RBAC: Authorizing `identity:create_grant(project_id=412274053ff24bdbbf7789817c7b575e, user_id=26969f9c613940a4b61f3afd6504eac7, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:45.336892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb16df0f-e9cc-46fc-b004-9c34ea00c99f admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:45.416870 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1309/2621] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:45 2022] PUT /identity/v3/projects/412274053ff24bdbbf7789817c7b575e/users/26969f9c613940a4b61f3afd6504eac7/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:45.427436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60f22d8-223d-4b86-a08b-8737ab750549 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:45.427714 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60f22d8-223d-4b86-a08b-8737ab750549 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:45.427982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60f22d8-223d-4b86-a08b-8737ab750549 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:45.461409 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-b60f22d8-223d-4b86-a08b-8737ab750549 None None] MFA Rules not processed for user `26969f9c613940a4b61f3afd6504eac7`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:45.462932 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-b60f22d8-223d-4b86-a08b-8737ab750549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:45.501557 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:45.558966 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1313/2622] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:45 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:45.570287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb16df0f-e9cc-46fc-b004-9c34ea00c99f admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:45.572325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-bb16df0f-e9cc-46fc-b004-9c34ea00c99f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:45.667709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb16df0f-e9cc-46fc-b004-9c34ea00c99f admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:45.672199 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7234bfcf-bf3c-459b-884c-e5a61a4f0f90 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:45.672853 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7234bfcf-bf3c-459b-884c-e5a61a4f0f90 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:45.673159 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7234bfcf-bf3c-459b-884c-e5a61a4f0f90 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:45.673428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7234bfcf-bf3c-459b-884c-e5a61a4f0f90 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:45.683692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7234bfcf-bf3c-459b-884c-e5a61a4f0f90 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:45.685352 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7234bfcf-bf3c-459b-884c-e5a61a4f0f90 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:45.701457 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1310/2623] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:45 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:45.725688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60f22d8-223d-4b86-a08b-8737ab750549 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:45.803682 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60f22d8-223d-4b86-a08b-8737ab750549 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:45.812043 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375d7346-e99f-45ee-9053-e88df78af54d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:45.812748 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375d7346-e99f-45ee-9053-e88df78af54d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:45.813027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375d7346-e99f-45ee-9053-e88df78af54d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:45.814601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375d7346-e99f-45ee-9053-e88df78af54d admin admin] PATH_INFO: `/v3/users/26969f9c613940a4b61f3afd6504eac7` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:45.816041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375d7346-e99f-45ee-9053-e88df78af54d admin admin] RBAC: Authorizing `identity:delete_user(user_id=26969f9c613940a4b61f3afd6504eac7)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:45.819048 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375d7346-e99f-45ee-9053-e88df78af54d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:46.017629 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-375d7346-e99f-45ee-9053-e88df78af54d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26969f9c613940a4b61f3afd6504eac7'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:46.025453 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-375d7346-e99f-45ee-9053-e88df78af54d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26969f9c613940a4b61f3afd6504eac7'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:46.041852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-375d7346-e99f-45ee-9053-e88df78af54d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26969f9c613940a4b61f3afd6504eac7'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:46.117317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-375d7346-e99f-45ee-9053-e88df78af54d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26969f9c613940a4b61f3afd6504eac7'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:46.118382 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-375d7346-e99f-45ee-9053-e88df78af54d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26969f9c613940a4b61f3afd6504eac7'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:46.125621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1314/2624] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:45 2022] DELETE /identity/v3/users/26969f9c613940a4b61f3afd6504eac7 => generated 0 bytes in 400 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:46.130158 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ba4637-0557-4f28-8561-627044069d7f None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:46.130643 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ba4637-0557-4f28-8561-627044069d7f None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:46.130950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ba4637-0557-4f28-8561-627044069d7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:46.152324 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-97ba4637-0557-4f28-8561-627044069d7f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:46.154896 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-97ba4637-0557-4f28-8561-627044069d7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:46.218880 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1311/2625] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:46.228624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375d7346-e99f-45ee-9053-e88df78af54d admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:46.230180 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-375d7346-e99f-45ee-9053-e88df78af54d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:46.304631 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375d7346-e99f-45ee-9053-e88df78af54d admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:46.309892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affbbe25-9be4-42e1-8bb2-18148083f10b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:46.310519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affbbe25-9be4-42e1-8bb2-18148083f10b service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:46.310805 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affbbe25-9be4-42e1-8bb2-18148083f10b service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:46.311069 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affbbe25-9be4-42e1-8bb2-18148083f10b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:46.318176 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:46.323850 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affbbe25-9be4-42e1-8bb2-18148083f10b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:46.325818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affbbe25-9be4-42e1-8bb2-18148083f10b service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:46.341594 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1315/2626] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:46.803800 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ba4637-0557-4f28-8561-627044069d7f None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:46.806029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-97ba4637-0557-4f28-8561-627044069d7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:46.818448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ba4637-0557-4f28-8561-627044069d7f None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:46.824020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef669eaa-d41a-431b-a192-e6e40ebacce7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:46.832370 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef669eaa-d41a-431b-a192-e6e40ebacce7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:46.832370 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef669eaa-d41a-431b-a192-e6e40ebacce7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:46.832370 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef669eaa-d41a-431b-a192-e6e40ebacce7 admin admin] PATH_INFO: `/v3/projects/412274053ff24bdbbf7789817c7b575e` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:46.832370 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef669eaa-d41a-431b-a192-e6e40ebacce7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=412274053ff24bdbbf7789817c7b575e)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:46.832370 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef669eaa-d41a-431b-a192-e6e40ebacce7 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:46.840188 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=94,keystone:DELETE=3,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:46.951434 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ef669eaa-d41a-431b-a192-e6e40ebacce7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '412274053ff24bdbbf7789817c7b575e', 'request_id': 'req-ef669eaa-d41a-431b-a192-e6e40ebacce7'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:46.956837 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ef669eaa-d41a-431b-a192-e6e40ebacce7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '412274053ff24bdbbf7789817c7b575e', 'request_id': 'req-ef669eaa-d41a-431b-a192-e6e40ebacce7'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:46.963661 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ef669eaa-d41a-431b-a192-e6e40ebacce7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '412274053ff24bdbbf7789817c7b575e', 'request_id': 'req-ef669eaa-d41a-431b-a192-e6e40ebacce7'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:47.030798 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ef669eaa-d41a-431b-a192-e6e40ebacce7 admin admin] The token cache is being invalidate because project 412274053ff24bdbbf7789817c7b575e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:47.031083 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ef669eaa-d41a-431b-a192-e6e40ebacce7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:47.031698 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ef669eaa-d41a-431b-a192-e6e40ebacce7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:47.033215 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1312/2627] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:46 2022] DELETE /identity/v3/projects/412274053ff24bdbbf7789817c7b575e => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:47.038174 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40052df9-9350-421d-8252-5acfe83e63af None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:47.039140 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40052df9-9350-421d-8252-5acfe83e63af None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:47.039277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40052df9-9350-421d-8252-5acfe83e63af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:47.059565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-40052df9-9350-421d-8252-5acfe83e63af None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:47.060948 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-40052df9-9350-421d-8252-5acfe83e63af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:47.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1316/2628] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:47.129906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a220ee-4ac2-4e77-8592-c4cd98e70139 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:47.132677 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a220ee-4ac2-4e77-8592-c4cd98e70139 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:47.132677 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a220ee-4ac2-4e77-8592-c4cd98e70139 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:47.150266 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-58a220ee-4ac2-4e77-8592-c4cd98e70139 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:47.151456 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-58a220ee-4ac2-4e77-8592-c4cd98e70139 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:47.159377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1313/2629] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:47.164499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40052df9-9350-421d-8252-5acfe83e63af None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:47.174216 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40052df9-9350-421d-8252-5acfe83e63af None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:47.178747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47053a4-02b0-4653-8fef-22d38142c6ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:47.179356 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47053a4-02b0-4653-8fef-22d38142c6ca admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:47.179632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47053a4-02b0-4653-8fef-22d38142c6ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:47.179899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47053a4-02b0-4653-8fef-22d38142c6ca admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:47.180563 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47053a4-02b0-4653-8fef-22d38142c6ca admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:33:47.180858 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47053a4-02b0-4653-8fef-22d38142c6ca admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:47.182422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47053a4-02b0-4653-8fef-22d38142c6ca admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:47.193812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1317/2630] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:33:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:33:47.198295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a220ee-4ac2-4e77-8592-c4cd98e70139 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:47.207891 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a220ee-4ac2-4e77-8592-c4cd98e70139 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:47.213560 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb7bc67-657a-40fa-a824-ffc46f0956b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:47.214155 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb7bc67-657a-40fa-a824-ffc46f0956b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:47.214427 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb7bc67-657a-40fa-a824-ffc46f0956b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:47.214734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb7bc67-657a-40fa-a824-ffc46f0956b8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:47.215354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb7bc67-657a-40fa-a824-ffc46f0956b8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:47.217259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb7bc67-657a-40fa-a824-ffc46f0956b8 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:47.301527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1314/2631] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:33:47 2022] POST /identity/v3/projects => generated 377 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:47.306360 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47053a4-02b0-4653-8fef-22d38142c6ca admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:47.316789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47053a4-02b0-4653-8fef-22d38142c6ca admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:47.374598 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f2653e-92dc-44d6-9fe1-16bd20801f2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:47.375134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f2653e-92dc-44d6-9fe1-16bd20801f2d admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:47.375408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f2653e-92dc-44d6-9fe1-16bd20801f2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:47.375678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f2653e-92dc-44d6-9fe1-16bd20801f2d admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:47.376333 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f2653e-92dc-44d6-9fe1-16bd20801f2d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:47.378259 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f2653e-92dc-44d6-9fe1-16bd20801f2d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:47.487639 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1318/2632] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:33:47 2022] POST /identity/v3/users => generated 289 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:47.492066 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb7bc67-657a-40fa-a824-ffc46f0956b8 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:47.502269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb7bc67-657a-40fa-a824-ffc46f0956b8 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:47.506865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ac4e52-a161-4b15-8825-6d2f15bccf1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:47.507951 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ac4e52-a161-4b15-8825-6d2f15bccf1e admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:47.508269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ac4e52-a161-4b15-8825-6d2f15bccf1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:47.508539 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ac4e52-a161-4b15-8825-6d2f15bccf1e admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:47.509169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ac4e52-a161-4b15-8825-6d2f15bccf1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:47.510780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ac4e52-a161-4b15-8825-6d2f15bccf1e admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:47.524482 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1315/2633] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:47.525727 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f2653e-92dc-44d6-9fe1-16bd20801f2d admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:47.535853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f2653e-92dc-44d6-9fe1-16bd20801f2d admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:47.540565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43597721-916e-4b0d-972a-92219c31ffe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:47.543233 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43597721-916e-4b0d-972a-92219c31ffe4 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:47.543233 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43597721-916e-4b0d-972a-92219c31ffe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:47.543233 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43597721-916e-4b0d-972a-92219c31ffe4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:47.543233 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43597721-916e-4b0d-972a-92219c31ffe4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:47.543977 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43597721-916e-4b0d-972a-92219c31ffe4 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:47.555028 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1319/2634] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:47.559122 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ac4e52-a161-4b15-8825-6d2f15bccf1e admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:47.569202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ac4e52-a161-4b15-8825-6d2f15bccf1e admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:47.574717 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4547a77a-064f-4694-9d8c-37c2ac80d97a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:47.575430 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4547a77a-064f-4694-9d8c-37c2ac80d97a admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:47.575778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4547a77a-064f-4694-9d8c-37c2ac80d97a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:47.576046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4547a77a-064f-4694-9d8c-37c2ac80d97a admin admin] PATH_INFO: `/v3/projects/71d2b2b87e5545aaafaeebdf113f91c2/users/d879e21b11984cb898aa3b7948e746c5/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:47.591823 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4547a77a-064f-4694-9d8c-37c2ac80d97a admin admin] RBAC: Authorizing `identity:create_grant(project_id=71d2b2b87e5545aaafaeebdf113f91c2, user_id=d879e21b11984cb898aa3b7948e746c5, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:47.594483 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4547a77a-064f-4694-9d8c-37c2ac80d97a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:47.659599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1316/2635] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:47 2022] PUT /identity/v3/projects/71d2b2b87e5545aaafaeebdf113f91c2/users/d879e21b11984cb898aa3b7948e746c5/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:33:47.670442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc567cf-c13f-4503-af1d-f89326c3b292 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:47.670801 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc567cf-c13f-4503-af1d-f89326c3b292 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:47.671148 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc567cf-c13f-4503-af1d-f89326c3b292 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:47.699337 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-fcc567cf-c13f-4503-af1d-f89326c3b292 None None] MFA Rules not processed for user `d879e21b11984cb898aa3b7948e746c5`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:47.700582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-fcc567cf-c13f-4503-af1d-f89326c3b292 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:47.749778 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:47.777358 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1320/2636] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:47 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:47.788774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4547a77a-064f-4694-9d8c-37c2ac80d97a admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:47.853410 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4547a77a-064f-4694-9d8c-37c2ac80d97a admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:47.860271 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818a52e8-7c49-4ba8-8e48-e65f98f61471 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:47.860892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a52e8-7c49-4ba8-8e48-e65f98f61471 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:47.861166 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a52e8-7c49-4ba8-8e48-e65f98f61471 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:47.861433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a52e8-7c49-4ba8-8e48-e65f98f61471 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:47.862061 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818a52e8-7c49-4ba8-8e48-e65f98f61471 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:47.864206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818a52e8-7c49-4ba8-8e48-e65f98f61471 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:47.953953 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1317/2637] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:33:47 2022] POST /identity/v3/projects => generated 375 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:47.958493 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc567cf-c13f-4503-af1d-f89326c3b292 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:47.969854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc567cf-c13f-4503-af1d-f89326c3b292 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:48.029381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f54c8b4-8aa6-4635-9326-5d6fb4b7a8de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:48.029993 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f54c8b4-8aa6-4635-9326-5d6fb4b7a8de admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:48.030520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f54c8b4-8aa6-4635-9326-5d6fb4b7a8de admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:48.030777 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f54c8b4-8aa6-4635-9326-5d6fb4b7a8de admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:48.031406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f54c8b4-8aa6-4635-9326-5d6fb4b7a8de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:48.033357 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f54c8b4-8aa6-4635-9326-5d6fb4b7a8de admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:48.106536 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1321/2638] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:33:47 2022] POST /identity/v3/users => generated 288 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:48.110448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818a52e8-7c49-4ba8-8e48-e65f98f61471 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:48.122960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818a52e8-7c49-4ba8-8e48-e65f98f61471 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:48.127888 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ab8e19-91f1-4d9f-9a5c-d92d9bacb3a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:48.128528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ab8e19-91f1-4d9f-9a5c-d92d9bacb3a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:48.128836 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ab8e19-91f1-4d9f-9a5c-d92d9bacb3a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:48.129107 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ab8e19-91f1-4d9f-9a5c-d92d9bacb3a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:48.129744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ab8e19-91f1-4d9f-9a5c-d92d9bacb3a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:48.131343 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ab8e19-91f1-4d9f-9a5c-d92d9bacb3a8 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:48.146810 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1318/2639] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:48 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:48.149047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f54c8b4-8aa6-4635-9326-5d6fb4b7a8de admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:48.159703 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f54c8b4-8aa6-4635-9326-5d6fb4b7a8de admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:48.166494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b57a4c-1d16-4ff2-9938-164e40e0ee16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:48.167195 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b57a4c-1d16-4ff2-9938-164e40e0ee16 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:48.167624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b57a4c-1d16-4ff2-9938-164e40e0ee16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:48.167958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b57a4c-1d16-4ff2-9938-164e40e0ee16 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:48.168794 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b57a4c-1d16-4ff2-9938-164e40e0ee16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:48.170627 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b57a4c-1d16-4ff2-9938-164e40e0ee16 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:48.181839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1322/2640] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:48 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:48.186054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ab8e19-91f1-4d9f-9a5c-d92d9bacb3a8 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:48.198307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ab8e19-91f1-4d9f-9a5c-d92d9bacb3a8 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:48.203153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31c9598-754b-4347-bfe8-c265443874cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:48.203807 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31c9598-754b-4347-bfe8-c265443874cc admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:48.204081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31c9598-754b-4347-bfe8-c265443874cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:48.204373 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31c9598-754b-4347-bfe8-c265443874cc admin admin] PATH_INFO: `/v3/projects/47f414e510274272828b36b3f3743d9a/users/95bab38a14d64475b708b498b6fdff82/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:48.208779 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:48.220273 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31c9598-754b-4347-bfe8-c265443874cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=47f414e510274272828b36b3f3743d9a, user_id=95bab38a14d64475b708b498b6fdff82, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:48.223119 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31c9598-754b-4347-bfe8-c265443874cc admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:48.300413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1319/2641] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:48 2022] PUT /identity/v3/projects/47f414e510274272828b36b3f3743d9a/users/95bab38a14d64475b708b498b6fdff82/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:48.311261 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a474ead-d84e-4d2b-b881-de3504ea5c3a None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:48.311539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a474ead-d84e-4d2b-b881-de3504ea5c3a None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:48.311808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a474ead-d84e-4d2b-b881-de3504ea5c3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:48.340387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-3a474ead-d84e-4d2b-b881-de3504ea5c3a None None] MFA Rules not processed for user `95bab38a14d64475b708b498b6fdff82`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:48.341595 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-3a474ead-d84e-4d2b-b881-de3504ea5c3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:48.411575 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1323/2642] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:48 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:48.416649 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31c9598-754b-4347-bfe8-c265443874cc admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:48.478612 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31c9598-754b-4347-bfe8-c265443874cc admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:48.485282 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2250b16b-a9ea-4d17-90f7-6738b2744eaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:48.485890 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2250b16b-a9ea-4d17-90f7-6738b2744eaf admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:48.486168 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2250b16b-a9ea-4d17-90f7-6738b2744eaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:48.486438 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2250b16b-a9ea-4d17-90f7-6738b2744eaf admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:48.487071 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2250b16b-a9ea-4d17-90f7-6738b2744eaf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:48.489074 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2250b16b-a9ea-4d17-90f7-6738b2744eaf admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:48.592691 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1320/2643] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:33:48 2022] POST /identity/v3/projects => generated 375 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:48.596772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a474ead-d84e-4d2b-b881-de3504ea5c3a None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:48.607975 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a474ead-d84e-4d2b-b881-de3504ea5c3a None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:48.660523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:48.667130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8310ab12-277c-40c7-8cce-de96455b43b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:48.667742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8310ab12-277c-40c7-8cce-de96455b43b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:48.668018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8310ab12-277c-40c7-8cce-de96455b43b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:48.668325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8310ab12-277c-40c7-8cce-de96455b43b1 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:48.668961 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8310ab12-277c-40c7-8cce-de96455b43b1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:48.670937 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8310ab12-277c-40c7-8cce-de96455b43b1 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:48.767531 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1324/2644] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:33:48 2022] POST /identity/v3/users => generated 294 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:33:48.771475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2250b16b-a9ea-4d17-90f7-6738b2744eaf admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:48.783546 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2250b16b-a9ea-4d17-90f7-6738b2744eaf admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:48.788982 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8688618-876a-4e1f-96d4-50afcfad6051 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:48.789556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8688618-876a-4e1f-96d4-50afcfad6051 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:48.789833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8688618-876a-4e1f-96d4-50afcfad6051 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:48.790100 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8688618-876a-4e1f-96d4-50afcfad6051 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:48.790726 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8688618-876a-4e1f-96d4-50afcfad6051 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:48.792332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8688618-876a-4e1f-96d4-50afcfad6051 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:48.802874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1321/2645] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:48 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:48.807198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8310ab12-277c-40c7-8cce-de96455b43b1 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:48.817379 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8310ab12-277c-40c7-8cce-de96455b43b1 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:48.822181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c840e372-d4ba-462f-8312-6268d9ab05be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:48.822879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c840e372-d4ba-462f-8312-6268d9ab05be admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:48.823129 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c840e372-d4ba-462f-8312-6268d9ab05be admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:48.823468 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c840e372-d4ba-462f-8312-6268d9ab05be admin admin] PATH_INFO: `/v3/projects/e9b9615310804d9b80cb706d31d27201/users/63b5615105b6439fbdfea2e8d29f94d6/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:48.839856 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c840e372-d4ba-462f-8312-6268d9ab05be admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9b9615310804d9b80cb706d31d27201, user_id=63b5615105b6439fbdfea2e8d29f94d6, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:48.842670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c840e372-d4ba-462f-8312-6268d9ab05be admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:48.946920 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1325/2646] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:48 2022] PUT /identity/v3/projects/e9b9615310804d9b80cb706d31d27201/users/63b5615105b6439fbdfea2e8d29f94d6/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:48.950941 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8688618-876a-4e1f-96d4-50afcfad6051 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:48.961695 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8688618-876a-4e1f-96d4-50afcfad6051 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:49.019067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b25bc3-9c18-4eca-84a4-a08162e2ab91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:49.019673 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b25bc3-9c18-4eca-84a4-a08162e2ab91 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:49.019951 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b25bc3-9c18-4eca-84a4-a08162e2ab91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:49.020392 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b25bc3-9c18-4eca-84a4-a08162e2ab91 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:49.021026 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b25bc3-9c18-4eca-84a4-a08162e2ab91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:49.022607 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b25bc3-9c18-4eca-84a4-a08162e2ab91 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:49.033369 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1322/2647] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:33:48 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:33:49.037566 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c840e372-d4ba-462f-8312-6268d9ab05be admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:49.047874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c840e372-d4ba-462f-8312-6268d9ab05be admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:49.052531 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308410a6-569d-4f27-8f7c-be921ccdbf67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:49.053336 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308410a6-569d-4f27-8f7c-be921ccdbf67 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:49.053611 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308410a6-569d-4f27-8f7c-be921ccdbf67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:49.053879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308410a6-569d-4f27-8f7c-be921ccdbf67 admin admin] PATH_INFO: `/v3/projects/e9b9615310804d9b80cb706d31d27201/users/63b5615105b6439fbdfea2e8d29f94d6/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:49.055791 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308410a6-569d-4f27-8f7c-be921ccdbf67 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9b9615310804d9b80cb706d31d27201, user_id=63b5615105b6439fbdfea2e8d29f94d6, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:49.058432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308410a6-569d-4f27-8f7c-be921ccdbf67 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:49.068520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1326/2648] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:33:49 2022] PUT /identity/v3/projects/e9b9615310804d9b80cb706d31d27201/users/63b5615105b6439fbdfea2e8d29f94d6/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:49.080192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76af3500-82f6-4443-bc19-04ed3498725d None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:49.080606 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76af3500-82f6-4443-bc19-04ed3498725d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:49.080953 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76af3500-82f6-4443-bc19-04ed3498725d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:49.109836 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-76af3500-82f6-4443-bc19-04ed3498725d None None] MFA Rules not processed for user `63b5615105b6439fbdfea2e8d29f94d6`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:49.111051 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-76af3500-82f6-4443-bc19-04ed3498725d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:49.156371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:49.181271 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1323/2649] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:49 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:49.193700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308410a6-569d-4f27-8f7c-be921ccdbf67 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:49.197105 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-308410a6-569d-4f27-8f7c-be921ccdbf67 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:49.255721 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308410a6-569d-4f27-8f7c-be921ccdbf67 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:49.261846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2564ff38-b8b1-4b87-a1e0-88d964006ca9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:49.262451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2564ff38-b8b1-4b87-a1e0-88d964006ca9 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:49.262728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2564ff38-b8b1-4b87-a1e0-88d964006ca9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:49.262997 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2564ff38-b8b1-4b87-a1e0-88d964006ca9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:49.320918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2564ff38-b8b1-4b87-a1e0-88d964006ca9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:49.322404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2564ff38-b8b1-4b87-a1e0-88d964006ca9 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:49.333524 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1327/2650] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:49 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:33:50.565363 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59677499-024b-4e72-a227-e21363575f9a None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:50.565913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59677499-024b-4e72-a227-e21363575f9a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:50.565913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59677499-024b-4e72-a227-e21363575f9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:50.585256 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-59677499-024b-4e72-a227-e21363575f9a None None] MFA Rules not processed for user `a91ac0ba7e8c4418930852b0fa184247`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:50.586613 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-59677499-024b-4e72-a227-e21363575f9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:50.642930 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1324/2651] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:50 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:51.273596 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2564ff38-b8b1-4b87-a1e0-88d964006ca9 service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:51.275143 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-2564ff38-b8b1-4b87-a1e0-88d964006ca9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:51.286708 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2564ff38-b8b1-4b87-a1e0-88d964006ca9 service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:51.290443 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771958c4-a780-4e82-861c-e9419e2c36e8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:51.291060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771958c4-a780-4e82-861c-e9419e2c36e8 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:51.291337 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771958c4-a780-4e82-861c-e9419e2c36e8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:51.291609 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771958c4-a780-4e82-861c-e9419e2c36e8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:51.358645 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771958c4-a780-4e82-861c-e9419e2c36e8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:51.360140 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771958c4-a780-4e82-861c-e9419e2c36e8 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:51.370208 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1328/2652] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:51 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:52.251951 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59677499-024b-4e72-a227-e21363575f9a None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:52.253510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-59677499-024b-4e72-a227-e21363575f9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:52.314120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59677499-024b-4e72-a227-e21363575f9a None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:52.317795 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7332aa09-7631-4bdf-89fb-d775530e3187 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:52.318414 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7332aa09-7631-4bdf-89fb-d775530e3187 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:52.318689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7332aa09-7631-4bdf-89fb-d775530e3187 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:52.318956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7332aa09-7631-4bdf-89fb-d775530e3187 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:52.325072 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7332aa09-7631-4bdf-89fb-d775530e3187 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:52.387513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7332aa09-7631-4bdf-89fb-d775530e3187 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:52.388986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7332aa09-7631-4bdf-89fb-d775530e3187 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:52.399201 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1325/2653] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:52 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:52.778678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771958c4-a780-4e82-861c-e9419e2c36e8 service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:52.783660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:52.788936 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771958c4-a780-4e82-861c-e9419e2c36e8 service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:52.793326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a55aa23-dab0-44ab-9dee-82c875bb7400 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:52.793456 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a55aa23-dab0-44ab-9dee-82c875bb7400 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:52.793841 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a55aa23-dab0-44ab-9dee-82c875bb7400 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:52.794122 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a55aa23-dab0-44ab-9dee-82c875bb7400 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:52.803714 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a55aa23-dab0-44ab-9dee-82c875bb7400 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:52.805361 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a55aa23-dab0-44ab-9dee-82c875bb7400 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:52.816086 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1329/2654] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:52 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:57.544142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7332aa09-7631-4bdf-89fb-d775530e3187 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:57.629548 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=100 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:57.654713 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7332aa09-7631-4bdf-89fb-d775530e3187 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:57.659860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdd06bc-f179-458f-891b-f2963c00d16f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:57.660583 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdd06bc-f179-458f-891b-f2963c00d16f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:57.660863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdd06bc-f179-458f-891b-f2963c00d16f admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:57.661186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdd06bc-f179-458f-891b-f2963c00d16f admin admin] PATH_INFO: `/v3/users/d879e21b11984cb898aa3b7948e746c5` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:57.663123 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdd06bc-f179-458f-891b-f2963c00d16f admin admin] RBAC: Authorizing `identity:delete_user(user_id=d879e21b11984cb898aa3b7948e746c5)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:57.665205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdd06bc-f179-458f-891b-f2963c00d16f admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:57.857688 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-3bdd06bc-f179-458f-891b-f2963c00d16f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd879e21b11984cb898aa3b7948e746c5'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:57.864533 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-3bdd06bc-f179-458f-891b-f2963c00d16f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd879e21b11984cb898aa3b7948e746c5'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:57.865339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-3bdd06bc-f179-458f-891b-f2963c00d16f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd879e21b11984cb898aa3b7948e746c5'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:57.878153 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-3bdd06bc-f179-458f-891b-f2963c00d16f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd879e21b11984cb898aa3b7948e746c5'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:57.974513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-3bdd06bc-f179-458f-891b-f2963c00d16f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd879e21b11984cb898aa3b7948e746c5'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:57.981634 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1326/2655] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:57 2022] DELETE /identity/v3/users/d879e21b11984cb898aa3b7948e746c5 => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:57.986177 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2c809a-9259-46f1-990c-a9e747efe5f7 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:57.986664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2c809a-9259-46f1-990c-a9e747efe5f7 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:57.986764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2c809a-9259-46f1-990c-a9e747efe5f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:58.006535 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-9c2c809a-9259-46f1-990c-a9e747efe5f7 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:33:58.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9c2c809a-9259-46f1-990c-a9e747efe5f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:58.072978 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1330/2656] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:33:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:33:58.083349 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdd06bc-f179-458f-891b-f2963c00d16f admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:58.084794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-3bdd06bc-f179-458f-891b-f2963c00d16f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:58.143376 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdd06bc-f179-458f-891b-f2963c00d16f admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:58.147428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1f92f1-a25d-45d5-afca-16ea3e80e5d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:58.148057 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1f92f1-a25d-45d5-afca-16ea3e80e5d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:58.148370 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1f92f1-a25d-45d5-afca-16ea3e80e5d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:58.148643 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1f92f1-a25d-45d5-afca-16ea3e80e5d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:58.158645 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1f92f1-a25d-45d5-afca-16ea3e80e5d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:58.160021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1f92f1-a25d-45d5-afca-16ea3e80e5d2 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:58.171926 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1327/2657] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:33:58.761729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2c809a-9259-46f1-990c-a9e747efe5f7 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:58.761729 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9c2c809a-9259-46f1-990c-a9e747efe5f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:58.774690 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2c809a-9259-46f1-990c-a9e747efe5f7 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:58.779859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6b5619-868a-4b47-8592-e672c56935c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:58.780602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b5619-868a-4b47-8592-e672c56935c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:58.780877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b5619-868a-4b47-8592-e672c56935c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:58.781150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b5619-868a-4b47-8592-e672c56935c1 admin admin] PATH_INFO: `/v3/projects/71d2b2b87e5545aaafaeebdf113f91c2` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:58.783097 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6b5619-868a-4b47-8592-e672c56935c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=71d2b2b87e5545aaafaeebdf113f91c2)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:58.784450 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6b5619-868a-4b47-8592-e672c56935c1 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:58.904550 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ad6b5619-868a-4b47-8592-e672c56935c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '71d2b2b87e5545aaafaeebdf113f91c2', 'request_id': 'req-ad6b5619-868a-4b47-8592-e672c56935c1'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:58.973709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ad6b5619-868a-4b47-8592-e672c56935c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '71d2b2b87e5545aaafaeebdf113f91c2', 'request_id': 'req-ad6b5619-868a-4b47-8592-e672c56935c1'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:58.979963 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ad6b5619-868a-4b47-8592-e672c56935c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '71d2b2b87e5545aaafaeebdf113f91c2', 'request_id': 'req-ad6b5619-868a-4b47-8592-e672c56935c1'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:58.991357 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ad6b5619-868a-4b47-8592-e672c56935c1 admin admin] The token cache is being invalidate because project 71d2b2b87e5545aaafaeebdf113f91c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:33:58.992178 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ad6b5619-868a-4b47-8592-e672c56935c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:58.992328 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-ad6b5619-868a-4b47-8592-e672c56935c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:58.993965 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1331/2658] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:33:58 2022] DELETE /identity/v3/projects/71d2b2b87e5545aaafaeebdf113f91c2 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:33:58.998083 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1f92f1-a25d-45d5-afca-16ea3e80e5d2 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:59.008533 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1f92f1-a25d-45d5-afca-16ea3e80e5d2 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:59.088629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dad674e-6e57-4c33-9d3d-7fa2f28cde1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:59.089660 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dad674e-6e57-4c33-9d3d-7fa2f28cde1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:59.090128 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dad674e-6e57-4c33-9d3d-7fa2f28cde1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:59.090590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dad674e-6e57-4c33-9d3d-7fa2f28cde1a admin admin] PATH_INFO: `/v3/users/95bab38a14d64475b708b498b6fdff82` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:59.092552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dad674e-6e57-4c33-9d3d-7fa2f28cde1a admin admin] RBAC: Authorizing `identity:delete_user(user_id=95bab38a14d64475b708b498b6fdff82)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:59.096086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dad674e-6e57-4c33-9d3d-7fa2f28cde1a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:59.099810 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:33:59.188534 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6b5619-868a-4b47-8592-e672c56935c1 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:33:59.190117 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-ad6b5619-868a-4b47-8592-e672c56935c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:59.249756 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6b5619-868a-4b47-8592-e672c56935c1 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:33:59.253365 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a93b73-c9fb-4a67-a992-058784c4b0ff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:33:59.253995 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a93b73-c9fb-4a67-a992-058784c4b0ff service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:33:59.254269 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a93b73-c9fb-4a67-a992-058784c4b0ff service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:33:59.254538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a93b73-c9fb-4a67-a992-058784c4b0ff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:33:59.256309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-a5a93b73-c9fb-4a67-a992-058784c4b0ff service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:33:59.269431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a93b73-c9fb-4a67-a992-058784c4b0ff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:33:59.270848 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a93b73-c9fb-4a67-a992-058784c4b0ff service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:33:59.280446 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1332/2659] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:33:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:33:59.322560 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-9dad674e-6e57-4c33-9d3d-7fa2f28cde1a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95bab38a14d64475b708b498b6fdff82'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:59.330503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-9dad674e-6e57-4c33-9d3d-7fa2f28cde1a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95bab38a14d64475b708b498b6fdff82'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:59.331221 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-9dad674e-6e57-4c33-9d3d-7fa2f28cde1a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95bab38a14d64475b708b498b6fdff82'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:59.348497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-9dad674e-6e57-4c33-9d3d-7fa2f28cde1a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95bab38a14d64475b708b498b6fdff82'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:59.431396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-9dad674e-6e57-4c33-9d3d-7fa2f28cde1a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95bab38a14d64475b708b498b6fdff82'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:33:59.438398 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1328/2660] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:33:58 2022] DELETE /identity/v3/users/95bab38a14d64475b708b498b6fdff82 => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:34:00.048258 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a93b73-c9fb-4a67-a992-058784c4b0ff service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:00.050237 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-a5a93b73-c9fb-4a67-a992-058784c4b0ff service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:00.062752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a93b73-c9fb-4a67-a992-058784c4b0ff service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:00.086832 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:00.170495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316370af-e137-4559-9126-298fe8f7c587 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:00.171569 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316370af-e137-4559-9126-298fe8f7c587 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:00.172273 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316370af-e137-4559-9126-298fe8f7c587 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:00.172891 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316370af-e137-4559-9126-298fe8f7c587 admin admin] PATH_INFO: `/v3/projects/47f414e510274272828b36b3f3743d9a` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:00.174659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316370af-e137-4559-9126-298fe8f7c587 admin admin] RBAC: Authorizing `identity:delete_project(project_id=47f414e510274272828b36b3f3743d9a)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:00.177608 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316370af-e137-4559-9126-298fe8f7c587 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:00.373715 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-316370af-e137-4559-9126-298fe8f7c587 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '47f414e510274272828b36b3f3743d9a', 'request_id': 'req-316370af-e137-4559-9126-298fe8f7c587'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:00.466846 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-316370af-e137-4559-9126-298fe8f7c587 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '47f414e510274272828b36b3f3743d9a', 'request_id': 'req-316370af-e137-4559-9126-298fe8f7c587'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:00.476761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-316370af-e137-4559-9126-298fe8f7c587 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '47f414e510274272828b36b3f3743d9a', 'request_id': 'req-316370af-e137-4559-9126-298fe8f7c587'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:00.488408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-316370af-e137-4559-9126-298fe8f7c587 admin admin] The token cache is being invalidate because project 47f414e510274272828b36b3f3743d9a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:34:00.488730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-316370af-e137-4559-9126-298fe8f7c587 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:00.489515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-316370af-e137-4559-9126-298fe8f7c587 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:00.491869 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1333/2661] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:34:00 2022] DELETE /identity/v3/projects/47f414e510274272828b36b3f3743d9a => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:00.495635 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dad674e-6e57-4c33-9d3d-7fa2f28cde1a admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:00.497203 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9dad674e-6e57-4c33-9d3d-7fa2f28cde1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:00.511350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dad674e-6e57-4c33-9d3d-7fa2f28cde1a admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:00.605788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad85ac5d-f4dd-413a-9823-9cd293306809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:00.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad85ac5d-f4dd-413a-9823-9cd293306809 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:00.606636 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad85ac5d-f4dd-413a-9823-9cd293306809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:00.606903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad85ac5d-f4dd-413a-9823-9cd293306809 admin admin] PATH_INFO: `/v3/users/63b5615105b6439fbdfea2e8d29f94d6` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:00.608426 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad85ac5d-f4dd-413a-9823-9cd293306809 admin admin] RBAC: Authorizing `identity:delete_user(user_id=63b5615105b6439fbdfea2e8d29f94d6)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:00.611182 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad85ac5d-f4dd-413a-9823-9cd293306809 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:00.822640 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ad85ac5d-f4dd-413a-9823-9cd293306809 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '63b5615105b6439fbdfea2e8d29f94d6'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:00.832844 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ad85ac5d-f4dd-413a-9823-9cd293306809 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '63b5615105b6439fbdfea2e8d29f94d6'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:00.833720 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ad85ac5d-f4dd-413a-9823-9cd293306809 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '63b5615105b6439fbdfea2e8d29f94d6'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:00.856631 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ad85ac5d-f4dd-413a-9823-9cd293306809 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '63b5615105b6439fbdfea2e8d29f94d6'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:00.945489 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ad85ac5d-f4dd-413a-9823-9cd293306809 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '63b5615105b6439fbdfea2e8d29f94d6'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:00.948350 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:00.954260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1329/2662] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:34:00 2022] DELETE /identity/v3/users/63b5615105b6439fbdfea2e8d29f94d6 => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:01.524725 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316370af-e137-4559-9126-298fe8f7c587 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:01.526309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-316370af-e137-4559-9126-298fe8f7c587 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:01.539912 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316370af-e137-4559-9126-298fe8f7c587 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:01.607378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7217fd8b-8a5b-4dcc-88e8-8cbd42699ff0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:01.609594 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7217fd8b-8a5b-4dcc-88e8-8cbd42699ff0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:01.609885 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7217fd8b-8a5b-4dcc-88e8-8cbd42699ff0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:01.610234 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7217fd8b-8a5b-4dcc-88e8-8cbd42699ff0 admin admin] PATH_INFO: `/v3/projects/e9b9615310804d9b80cb706d31d27201` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:01.611422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7217fd8b-8a5b-4dcc-88e8-8cbd42699ff0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e9b9615310804d9b80cb706d31d27201)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:01.613609 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7217fd8b-8a5b-4dcc-88e8-8cbd42699ff0 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:01.714583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:01.715854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7217fd8b-8a5b-4dcc-88e8-8cbd42699ff0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e9b9615310804d9b80cb706d31d27201', 'request_id': 'req-7217fd8b-8a5b-4dcc-88e8-8cbd42699ff0'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:01.763293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7217fd8b-8a5b-4dcc-88e8-8cbd42699ff0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e9b9615310804d9b80cb706d31d27201', 'request_id': 'req-7217fd8b-8a5b-4dcc-88e8-8cbd42699ff0'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:01.769558 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7217fd8b-8a5b-4dcc-88e8-8cbd42699ff0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e9b9615310804d9b80cb706d31d27201', 'request_id': 'req-7217fd8b-8a5b-4dcc-88e8-8cbd42699ff0'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:01.781049 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7217fd8b-8a5b-4dcc-88e8-8cbd42699ff0 admin admin] The token cache is being invalidate because project e9b9615310804d9b80cb706d31d27201 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:34:01.781700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7217fd8b-8a5b-4dcc-88e8-8cbd42699ff0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:01.782421 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-7217fd8b-8a5b-4dcc-88e8-8cbd42699ff0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:01.784168 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1334/2663] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:34:01 2022] DELETE /identity/v3/projects/e9b9615310804d9b80cb706d31d27201 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:01.789445 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923be749-a5eb-4ef8-9d54-7f54cc9f4fe4 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:01.790022 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923be749-a5eb-4ef8-9d54-7f54cc9f4fe4 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:01.790134 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923be749-a5eb-4ef8-9d54-7f54cc9f4fe4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:01.814643 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-923be749-a5eb-4ef8-9d54-7f54cc9f4fe4 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:34:01.816767 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-923be749-a5eb-4ef8-9d54-7f54cc9f4fe4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:01.882020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1330/2664] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:34:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:34:01.889862 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365eb75b-440d-488c-9bb7-ceffa9d55c43 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:01.890408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365eb75b-440d-488c-9bb7-ceffa9d55c43 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:01.891021 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365eb75b-440d-488c-9bb7-ceffa9d55c43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:01.910591 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-365eb75b-440d-488c-9bb7-ceffa9d55c43 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:34:01.911788 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-365eb75b-440d-488c-9bb7-ceffa9d55c43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:01.921058 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1335/2665] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:34:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:34:01.925128 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923be749-a5eb-4ef8-9d54-7f54cc9f4fe4 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:01.936218 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923be749-a5eb-4ef8-9d54-7f54cc9f4fe4 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:01.941062 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7029acb-af47-4abf-bd53-622662602a1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:01.941670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7029acb-af47-4abf-bd53-622662602a1e admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:01.941947 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7029acb-af47-4abf-bd53-622662602a1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:01.942216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7029acb-af47-4abf-bd53-622662602a1e admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:01.942860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7029acb-af47-4abf-bd53-622662602a1e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:34:01.943167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7029acb-af47-4abf-bd53-622662602a1e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:01.944880 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7029acb-af47-4abf-bd53-622662602a1e admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:01.960945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1331/2666] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:34:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:34:01.966452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365eb75b-440d-488c-9bb7-ceffa9d55c43 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:01.976499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365eb75b-440d-488c-9bb7-ceffa9d55c43 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:01.981242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65579b58-e190-4503-a72e-f6585a1ad289 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:01.981866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65579b58-e190-4503-a72e-f6585a1ad289 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:01.982120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65579b58-e190-4503-a72e-f6585a1ad289 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:01.982389 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65579b58-e190-4503-a72e-f6585a1ad289 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:01.983147 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65579b58-e190-4503-a72e-f6585a1ad289 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:01.985118 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65579b58-e190-4503-a72e-f6585a1ad289 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:02.066008 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1336/2667] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:34:01 2022] POST /identity/v3/projects => generated 385 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:34:02.067131 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7029acb-af47-4abf-bd53-622662602a1e admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:02.078551 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7029acb-af47-4abf-bd53-622662602a1e admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:02.170380 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538acad9-83e8-47f7-9dd0-da12592938d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:02.171216 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538acad9-83e8-47f7-9dd0-da12592938d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:02.171601 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538acad9-83e8-47f7-9dd0-da12592938d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:02.171920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538acad9-83e8-47f7-9dd0-da12592938d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:02.172757 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538acad9-83e8-47f7-9dd0-da12592938d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:02.175103 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538acad9-83e8-47f7-9dd0-da12592938d4 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:02.263443 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1332/2668] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:34:02 2022] POST /identity/v3/users => generated 293 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:34:02.267572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65579b58-e190-4503-a72e-f6585a1ad289 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:02.291330 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65579b58-e190-4503-a72e-f6585a1ad289 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:02.316639 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6005b9b-73db-43e3-af34-d1bb82965160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:02.317264 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6005b9b-73db-43e3-af34-d1bb82965160 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:02.317539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6005b9b-73db-43e3-af34-d1bb82965160 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:02.317805 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6005b9b-73db-43e3-af34-d1bb82965160 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:02.318787 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6005b9b-73db-43e3-af34-d1bb82965160 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:02.320574 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6005b9b-73db-43e3-af34-d1bb82965160 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:02.341913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538acad9-83e8-47f7-9dd0-da12592938d4 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:02.344542 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1337/2669] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:34:02 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:34:02.352956 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538acad9-83e8-47f7-9dd0-da12592938d4 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:02.358146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1adc06-fb19-4bae-96b4-c46264b9f1e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:02.358541 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1adc06-fb19-4bae-96b4-c46264b9f1e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:02.358818 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1adc06-fb19-4bae-96b4-c46264b9f1e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:02.359082 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1adc06-fb19-4bae-96b4-c46264b9f1e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:02.359722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1adc06-fb19-4bae-96b4-c46264b9f1e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:02.361687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1adc06-fb19-4bae-96b4-c46264b9f1e5 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:02.373335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1333/2670] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:34:02 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:34:02.379895 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6005b9b-73db-43e3-af34-d1bb82965160 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:02.403994 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6005b9b-73db-43e3-af34-d1bb82965160 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:02.416731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32be7b3-8001-4e36-96c6-1aa567834fe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:02.418342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32be7b3-8001-4e36-96c6-1aa567834fe6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:02.418735 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32be7b3-8001-4e36-96c6-1aa567834fe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:02.419609 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32be7b3-8001-4e36-96c6-1aa567834fe6 admin admin] PATH_INFO: `/v3/projects/6ea5e68c0fd84fc9b4e02d6b4d391918/users/2d4429ab15b3492e94a84feff64ab6e2/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:02.461212 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32be7b3-8001-4e36-96c6-1aa567834fe6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ea5e68c0fd84fc9b4e02d6b4d391918, user_id=2d4429ab15b3492e94a84feff64ab6e2, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:02.463976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32be7b3-8001-4e36-96c6-1aa567834fe6 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:02.578377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1338/2671] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:34:02 2022] PUT /identity/v3/projects/6ea5e68c0fd84fc9b4e02d6b4d391918/users/2d4429ab15b3492e94a84feff64ab6e2/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:02.638338 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8115ce0-38f1-4e00-8ca2-2f3c37b26117 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:02.638613 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8115ce0-38f1-4e00-8ca2-2f3c37b26117 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:02.638894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8115ce0-38f1-4e00-8ca2-2f3c37b26117 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:02.685054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-d8115ce0-38f1-4e00-8ca2-2f3c37b26117 None None] MFA Rules not processed for user `2d4429ab15b3492e94a84feff64ab6e2`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:34:02.686681 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d8115ce0-38f1-4e00-8ca2-2f3c37b26117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:02.767268 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:02.779625 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1334/2672] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:34:02 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:34:02.784033 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32be7b3-8001-4e36-96c6-1aa567834fe6 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:02.879015 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32be7b3-8001-4e36-96c6-1aa567834fe6 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:02.887515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1133191-8338-4fa5-b997-d7548ab9b936 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:02.888420 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1133191-8338-4fa5-b997-d7548ab9b936 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:02.888883 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1133191-8338-4fa5-b997-d7548ab9b936 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:02.889334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1133191-8338-4fa5-b997-d7548ab9b936 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:02.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1133191-8338-4fa5-b997-d7548ab9b936 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:02.898408 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1133191-8338-4fa5-b997-d7548ab9b936 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:03.030174 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1339/2673] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:34:02 2022] POST /identity/v3/projects => generated 385 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:34:03.031433 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8115ce0-38f1-4e00-8ca2-2f3c37b26117 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:03.043374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8115ce0-38f1-4e00-8ca2-2f3c37b26117 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:03.107787 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea85bf45-42fb-4598-8334-00631a12c480 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:03.108827 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea85bf45-42fb-4598-8334-00631a12c480 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:03.109326 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea85bf45-42fb-4598-8334-00631a12c480 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:03.109665 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea85bf45-42fb-4598-8334-00631a12c480 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:03.110390 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea85bf45-42fb-4598-8334-00631a12c480 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:03.112394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea85bf45-42fb-4598-8334-00631a12c480 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:03.247833 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1335/2674] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:34:03 2022] POST /identity/v3/users => generated 293 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:34:03.247949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1133191-8338-4fa5-b997-d7548ab9b936 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:03.259829 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1133191-8338-4fa5-b997-d7548ab9b936 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:03.264828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159f0b10-8069-492d-910b-de7d40f9db93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:03.265510 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159f0b10-8069-492d-910b-de7d40f9db93 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:03.265852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159f0b10-8069-492d-910b-de7d40f9db93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:03.266292 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159f0b10-8069-492d-910b-de7d40f9db93 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:03.267017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159f0b10-8069-492d-910b-de7d40f9db93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:03.271715 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159f0b10-8069-492d-910b-de7d40f9db93 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:03.283697 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1340/2675] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:34:03 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:34:03.289108 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea85bf45-42fb-4598-8334-00631a12c480 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:03.299211 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea85bf45-42fb-4598-8334-00631a12c480 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:03.304509 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43690b8c-b88e-4695-8792-bc166defa733 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:03.305105 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43690b8c-b88e-4695-8792-bc166defa733 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:03.305375 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43690b8c-b88e-4695-8792-bc166defa733 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:03.305647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43690b8c-b88e-4695-8792-bc166defa733 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:03.306275 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43690b8c-b88e-4695-8792-bc166defa733 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:03.307865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43690b8c-b88e-4695-8792-bc166defa733 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:03.319206 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1336/2676] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:34:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:34:03.324276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159f0b10-8069-492d-910b-de7d40f9db93 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:03.336274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159f0b10-8069-492d-910b-de7d40f9db93 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:03.340277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a8350a-a95b-45e0-bd3f-868a70e24ba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:03.341290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a8350a-a95b-45e0-bd3f-868a70e24ba4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:03.341290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a8350a-a95b-45e0-bd3f-868a70e24ba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:03.341539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a8350a-a95b-45e0-bd3f-868a70e24ba4 admin admin] PATH_INFO: `/v3/projects/91082b44fa08489dab1cc89e3cff5a9e/users/d8231a41df9b4646bda7fa2ef022715c/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:03.359460 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a8350a-a95b-45e0-bd3f-868a70e24ba4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=91082b44fa08489dab1cc89e3cff5a9e, user_id=d8231a41df9b4646bda7fa2ef022715c, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:03.362299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a8350a-a95b-45e0-bd3f-868a70e24ba4 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:03.455419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1341/2677] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:34:03 2022] PUT /identity/v3/projects/91082b44fa08489dab1cc89e3cff5a9e/users/d8231a41df9b4646bda7fa2ef022715c/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:03.466247 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0008e7e-d6da-44a9-a262-593dd6c6f9ef None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:03.466636 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0008e7e-d6da-44a9-a262-593dd6c6f9ef None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:03.466819 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0008e7e-d6da-44a9-a262-593dd6c6f9ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:03.494762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-b0008e7e-d6da-44a9-a262-593dd6c6f9ef None None] MFA Rules not processed for user `d8231a41df9b4646bda7fa2ef022715c`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:34:03.495919 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b0008e7e-d6da-44a9-a262-593dd6c6f9ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:03.573813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1337/2678] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:34:03 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:34:03.575318 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a8350a-a95b-45e0-bd3f-868a70e24ba4 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:03.609817 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:03.652041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a8350a-a95b-45e0-bd3f-868a70e24ba4 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:03.660570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db267fe9-6cb0-4698-a85b-3127911c7468 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:03.661196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db267fe9-6cb0-4698-a85b-3127911c7468 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:03.661471 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db267fe9-6cb0-4698-a85b-3127911c7468 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:03.661739 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db267fe9-6cb0-4698-a85b-3127911c7468 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:03.662369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db267fe9-6cb0-4698-a85b-3127911c7468 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:03.664907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db267fe9-6cb0-4698-a85b-3127911c7468 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:03.797371 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1342/2679] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:34:03 2022] POST /identity/v3/projects => generated 385 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:34:03.798887 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0008e7e-d6da-44a9-a262-593dd6c6f9ef None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:03.811071 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0008e7e-d6da-44a9-a262-593dd6c6f9ef None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:03.873677 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84682168-4df2-4b6e-964d-45cd37ef8d35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:03.874406 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84682168-4df2-4b6e-964d-45cd37ef8d35 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:03.874686 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84682168-4df2-4b6e-964d-45cd37ef8d35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:03.874958 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84682168-4df2-4b6e-964d-45cd37ef8d35 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:03.875593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84682168-4df2-4b6e-964d-45cd37ef8d35 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:03.881388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84682168-4df2-4b6e-964d-45cd37ef8d35 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:03.888492 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:03.974912 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1338/2680] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:34:03 2022] POST /identity/v3/users => generated 299 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:34:03.979804 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db267fe9-6cb0-4698-a85b-3127911c7468 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:03.990589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db267fe9-6cb0-4698-a85b-3127911c7468 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:03.996721 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c477d97c-09d2-48f3-88ad-17d62ba8c0ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:03.997386 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c477d97c-09d2-48f3-88ad-17d62ba8c0ab admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:03.997731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c477d97c-09d2-48f3-88ad-17d62ba8c0ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:03.998074 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c477d97c-09d2-48f3-88ad-17d62ba8c0ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:03.998779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c477d97c-09d2-48f3-88ad-17d62ba8c0ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:04.000508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c477d97c-09d2-48f3-88ad-17d62ba8c0ab admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:04.012899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1343/2681] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:34:03 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:34:04.016983 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84682168-4df2-4b6e-964d-45cd37ef8d35 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:04.027899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84682168-4df2-4b6e-964d-45cd37ef8d35 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:04.032994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696001a6-af31-463c-aa75-fe94b4a2bed8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:04.033780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696001a6-af31-463c-aa75-fe94b4a2bed8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:04.034133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696001a6-af31-463c-aa75-fe94b4a2bed8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:04.034477 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696001a6-af31-463c-aa75-fe94b4a2bed8 admin admin] PATH_INFO: `/v3/projects/ad0d434b186f49a98eaf787d403f37cd/users/a790ec68cd674211b394905a98765940/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:04.050504 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696001a6-af31-463c-aa75-fe94b4a2bed8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad0d434b186f49a98eaf787d403f37cd, user_id=a790ec68cd674211b394905a98765940, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:04.053258 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696001a6-af31-463c-aa75-fe94b4a2bed8 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:04.144017 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1339/2682] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:34:04 2022] PUT /identity/v3/projects/ad0d434b186f49a98eaf787d403f37cd/users/a790ec68cd674211b394905a98765940/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:04.149071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c477d97c-09d2-48f3-88ad-17d62ba8c0ab admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:04.168075 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c477d97c-09d2-48f3-88ad-17d62ba8c0ab admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:04.237885 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd33a1b-325c-4a95-be69-7312c2ce117a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:04.238622 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd33a1b-325c-4a95-be69-7312c2ce117a admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:04.239017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd33a1b-325c-4a95-be69-7312c2ce117a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:04.239393 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd33a1b-325c-4a95-be69-7312c2ce117a admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:04.240175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd33a1b-325c-4a95-be69-7312c2ce117a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:04.242949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd33a1b-325c-4a95-be69-7312c2ce117a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:04.255807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1344/2683] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:34:04 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:34:04.259533 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696001a6-af31-463c-aa75-fe94b4a2bed8 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:04.271116 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696001a6-af31-463c-aa75-fe94b4a2bed8 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:04.278492 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f98f66-f17b-48a6-9e07-f1572748520c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:04.279280 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f98f66-f17b-48a6-9e07-f1572748520c admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:04.279631 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f98f66-f17b-48a6-9e07-f1572748520c admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:04.280600 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f98f66-f17b-48a6-9e07-f1572748520c admin admin] PATH_INFO: `/v3/projects/ad0d434b186f49a98eaf787d403f37cd/users/a790ec68cd674211b394905a98765940/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:04.282778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f98f66-f17b-48a6-9e07-f1572748520c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad0d434b186f49a98eaf787d403f37cd, user_id=a790ec68cd674211b394905a98765940, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:04.286392 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f98f66-f17b-48a6-9e07-f1572748520c admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:04.308070 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1340/2684] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:34:04 2022] PUT /identity/v3/projects/ad0d434b186f49a98eaf787d403f37cd/users/a790ec68cd674211b394905a98765940/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:34:04.319363 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a6c223-4706-4555-a29c-5d112f0bce22 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:04.320051 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a6c223-4706-4555-a29c-5d112f0bce22 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:04.320601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a6c223-4706-4555-a29c-5d112f0bce22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:04.364060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-99a6c223-4706-4555-a29c-5d112f0bce22 None None] MFA Rules not processed for user `a790ec68cd674211b394905a98765940`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:34:04.366253 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-99a6c223-4706-4555-a29c-5d112f0bce22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:04.443335 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1345/2685] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:34:04 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:34:04.457111 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f98f66-f17b-48a6-9e07-f1572748520c admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:04.459130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-19f98f66-f17b-48a6-9e07-f1572748520c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:04.555661 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f98f66-f17b-48a6-9e07-f1572748520c admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:04.559707 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dd1064-093b-4e6e-a2b8-a92535d6e70f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:04.560367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dd1064-093b-4e6e-a2b8-a92535d6e70f service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:04.560700 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dd1064-093b-4e6e-a2b8-a92535d6e70f service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:04.560980 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dd1064-093b-4e6e-a2b8-a92535d6e70f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:04.632129 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6dd1064-093b-4e6e-a2b8-a92535d6e70f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:04.633834 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6dd1064-093b-4e6e-a2b8-a92535d6e70f service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:04.646133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1341/2686] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:34:04 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:34:06.708452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a6c223-4706-4555-a29c-5d112f0bce22 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:06.709876 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-99a6c223-4706-4555-a29c-5d112f0bce22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:06.721539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a6c223-4706-4555-a29c-5d112f0bce22 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:06.725322 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fab60a-f619-4210-8bd5-2ae82c973c71 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:06.725920 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fab60a-f619-4210-8bd5-2ae82c973c71 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:06.726197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fab60a-f619-4210-8bd5-2ae82c973c71 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:06.726463 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fab60a-f619-4210-8bd5-2ae82c973c71 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:06.740494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:06.795092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fab60a-f619-4210-8bd5-2ae82c973c71 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:06.796585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fab60a-f619-4210-8bd5-2ae82c973c71 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:06.809464 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1346/2687] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:34:06 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:34:08.308306 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dd1064-093b-4e6e-a2b8-a92535d6e70f service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:08.310906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f6dd1064-093b-4e6e-a2b8-a92535d6e70f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:08.417200 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dd1064-093b-4e6e-a2b8-a92535d6e70f service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:08.424667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cce5ffb-5a58-4c33-b988-43993c25b968 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:08.426001 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cce5ffb-5a58-4c33-b988-43993c25b968 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:08.426472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cce5ffb-5a58-4c33-b988-43993c25b968 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:08.426933 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cce5ffb-5a58-4c33-b988-43993c25b968 admin admin] PATH_INFO: `/v3/users/03d34d1e66404d259599806d040628ef` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:08.429464 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cce5ffb-5a58-4c33-b988-43993c25b968 admin admin] RBAC: Authorizing `identity:delete_user(user_id=03d34d1e66404d259599806d040628ef)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:08.432984 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cce5ffb-5a58-4c33-b988-43993c25b968 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:08.458048 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:INSERT=3,keystone:SELECT=96,keystone:DELETE=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:08.663574 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7cce5ffb-5a58-4c33-b988-43993c25b968 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '03d34d1e66404d259599806d040628ef'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:08.669928 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7cce5ffb-5a58-4c33-b988-43993c25b968 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '03d34d1e66404d259599806d040628ef'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:08.670636 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7cce5ffb-5a58-4c33-b988-43993c25b968 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '03d34d1e66404d259599806d040628ef'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:08.673399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fab60a-f619-4210-8bd5-2ae82c973c71 service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:08.683306 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fab60a-f619-4210-8bd5-2ae82c973c71 service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:08.687243 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10da4d0-e284-4756-9358-70358036e94c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:08.688089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10da4d0-e284-4756-9358-70358036e94c service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:08.688401 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10da4d0-e284-4756-9358-70358036e94c service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:08.688576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7cce5ffb-5a58-4c33-b988-43993c25b968 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '03d34d1e66404d259599806d040628ef'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:08.688749 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10da4d0-e284-4756-9358-70358036e94c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:08.699531 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10da4d0-e284-4756-9358-70358036e94c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:08.703870 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10da4d0-e284-4756-9358-70358036e94c service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:08.719538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1347/2688] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:34:08 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:34:08.767025 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-7cce5ffb-5a58-4c33-b988-43993c25b968 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '03d34d1e66404d259599806d040628ef'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:08.769905 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1342/2689] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:34:08 2022] DELETE /identity/v3/users/03d34d1e66404d259599806d040628ef => generated 0 bytes in 463 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:09.032493 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10da4d0-e284-4756-9358-70358036e94c service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:09.044131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10da4d0-e284-4756-9358-70358036e94c service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:09.106332 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6222618-311e-46eb-903a-7b98fc46c116 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:09.106960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6222618-311e-46eb-903a-7b98fc46c116 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:09.107234 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6222618-311e-46eb-903a-7b98fc46c116 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:09.107504 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6222618-311e-46eb-903a-7b98fc46c116 admin admin] PATH_INFO: `/v3/projects/ac06ecc6fc93477ba4909284525df514` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:09.108653 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6222618-311e-46eb-903a-7b98fc46c116 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ac06ecc6fc93477ba4909284525df514)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:09.110640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6222618-311e-46eb-903a-7b98fc46c116 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:09.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-a6222618-311e-46eb-903a-7b98fc46c116 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ac06ecc6fc93477ba4909284525df514', 'request_id': 'req-a6222618-311e-46eb-903a-7b98fc46c116'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:09.392047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-a6222618-311e-46eb-903a-7b98fc46c116 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ac06ecc6fc93477ba4909284525df514', 'request_id': 'req-a6222618-311e-46eb-903a-7b98fc46c116'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:09.398478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-a6222618-311e-46eb-903a-7b98fc46c116 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ac06ecc6fc93477ba4909284525df514', 'request_id': 'req-a6222618-311e-46eb-903a-7b98fc46c116'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:09.409774 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-a6222618-311e-46eb-903a-7b98fc46c116 admin admin] The token cache is being invalidate because project ac06ecc6fc93477ba4909284525df514 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:34:09.410061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-a6222618-311e-46eb-903a-7b98fc46c116 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:09.410709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-a6222618-311e-46eb-903a-7b98fc46c116 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:09.412398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1348/2690] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:34:09 2022] DELETE /identity/v3/projects/ac06ecc6fc93477ba4909284525df514 => generated 0 bytes in 381 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:09.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cce5ffb-5a58-4c33-b988-43993c25b968 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:09.418636 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7cce5ffb-5a58-4c33-b988-43993c25b968 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:09.432841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cce5ffb-5a58-4c33-b988-43993c25b968 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:09.489632 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad79cd29-5052-4572-973d-2b8f7f5b7ac8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:09.490277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad79cd29-5052-4572-973d-2b8f7f5b7ac8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:09.490544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad79cd29-5052-4572-973d-2b8f7f5b7ac8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:09.490871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad79cd29-5052-4572-973d-2b8f7f5b7ac8 admin admin] PATH_INFO: `/v3/users/8717d3e589424f00abdfcc77395a9db8` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:09.492670 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad79cd29-5052-4572-973d-2b8f7f5b7ac8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8717d3e589424f00abdfcc77395a9db8)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:09.494853 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad79cd29-5052-4572-973d-2b8f7f5b7ac8 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:09.661117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ad79cd29-5052-4572-973d-2b8f7f5b7ac8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8717d3e589424f00abdfcc77395a9db8'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:09.669345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ad79cd29-5052-4572-973d-2b8f7f5b7ac8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8717d3e589424f00abdfcc77395a9db8'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:09.670080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ad79cd29-5052-4572-973d-2b8f7f5b7ac8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8717d3e589424f00abdfcc77395a9db8'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:09.686642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ad79cd29-5052-4572-973d-2b8f7f5b7ac8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8717d3e589424f00abdfcc77395a9db8'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:09.882037 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ad79cd29-5052-4572-973d-2b8f7f5b7ac8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8717d3e589424f00abdfcc77395a9db8'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:09.888831 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1343/2691] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:34:09 2022] DELETE /identity/v3/users/8717d3e589424f00abdfcc77395a9db8 => generated 0 bytes in 472 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:10.148315 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6222618-311e-46eb-903a-7b98fc46c116 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:10.149889 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-a6222618-311e-46eb-903a-7b98fc46c116 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:10.163054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6222618-311e-46eb-903a-7b98fc46c116 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:10.194957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:10.221378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfd63bb-d118-4b1c-9dc0-f17c0cb2f34c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:10.222019 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfd63bb-d118-4b1c-9dc0-f17c0cb2f34c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:10.222289 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfd63bb-d118-4b1c-9dc0-f17c0cb2f34c admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:10.222555 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfd63bb-d118-4b1c-9dc0-f17c0cb2f34c admin admin] PATH_INFO: `/v3/projects/df1eeabcc1264062a10d42671ad56c3c` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:10.223685 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfd63bb-d118-4b1c-9dc0-f17c0cb2f34c admin admin] RBAC: Authorizing `identity:delete_project(project_id=df1eeabcc1264062a10d42671ad56c3c)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:10.225634 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfd63bb-d118-4b1c-9dc0-f17c0cb2f34c admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:10.380224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2bfd63bb-d118-4b1c-9dc0-f17c0cb2f34c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df1eeabcc1264062a10d42671ad56c3c', 'request_id': 'req-2bfd63bb-d118-4b1c-9dc0-f17c0cb2f34c'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:10.506103 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2bfd63bb-d118-4b1c-9dc0-f17c0cb2f34c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df1eeabcc1264062a10d42671ad56c3c', 'request_id': 'req-2bfd63bb-d118-4b1c-9dc0-f17c0cb2f34c'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:10.513048 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2bfd63bb-d118-4b1c-9dc0-f17c0cb2f34c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df1eeabcc1264062a10d42671ad56c3c', 'request_id': 'req-2bfd63bb-d118-4b1c-9dc0-f17c0cb2f34c'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:10.524236 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2bfd63bb-d118-4b1c-9dc0-f17c0cb2f34c admin admin] The token cache is being invalidate because project df1eeabcc1264062a10d42671ad56c3c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:34:10.524525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2bfd63bb-d118-4b1c-9dc0-f17c0cb2f34c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:10.525423 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2bfd63bb-d118-4b1c-9dc0-f17c0cb2f34c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:10.526917 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1349/2692] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:34:10 2022] DELETE /identity/v3/projects/df1eeabcc1264062a10d42671ad56c3c => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:13.448797 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad79cd29-5052-4572-973d-2b8f7f5b7ac8 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:13.450276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ad79cd29-5052-4572-973d-2b8f7f5b7ac8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:13.464570 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad79cd29-5052-4572-973d-2b8f7f5b7ac8 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:13.470648 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:13.540826 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeaa2d01-a99d-4d2c-9ad2-5d11bf8a0ea0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:13.541472 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeaa2d01-a99d-4d2c-9ad2-5d11bf8a0ea0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:13.541762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeaa2d01-a99d-4d2c-9ad2-5d11bf8a0ea0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:13.542021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeaa2d01-a99d-4d2c-9ad2-5d11bf8a0ea0 admin admin] PATH_INFO: `/v3/users/2d4429ab15b3492e94a84feff64ab6e2` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:13.543175 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeaa2d01-a99d-4d2c-9ad2-5d11bf8a0ea0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d4429ab15b3492e94a84feff64ab6e2)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:13.545235 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeaa2d01-a99d-4d2c-9ad2-5d11bf8a0ea0 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:13.810463 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-eeaa2d01-a99d-4d2c-9ad2-5d11bf8a0ea0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d4429ab15b3492e94a84feff64ab6e2'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:13.818967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-eeaa2d01-a99d-4d2c-9ad2-5d11bf8a0ea0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d4429ab15b3492e94a84feff64ab6e2'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:13.820184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-eeaa2d01-a99d-4d2c-9ad2-5d11bf8a0ea0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d4429ab15b3492e94a84feff64ab6e2'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:13.833667 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-eeaa2d01-a99d-4d2c-9ad2-5d11bf8a0ea0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d4429ab15b3492e94a84feff64ab6e2'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:13.923194 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-eeaa2d01-a99d-4d2c-9ad2-5d11bf8a0ea0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d4429ab15b3492e94a84feff64ab6e2'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:13.929867 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1344/2693] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:34:13 2022] DELETE /identity/v3/users/2d4429ab15b3492e94a84feff64ab6e2 => generated 0 bytes in 482 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:13.934989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be62555-02a2-4c60-9e3b-517c794979d6 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:13.935866 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be62555-02a2-4c60-9e3b-517c794979d6 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:13.936077 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be62555-02a2-4c60-9e3b-517c794979d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:13.957827 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-6be62555-02a2-4c60-9e3b-517c794979d6 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:34:13.959045 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6be62555-02a2-4c60-9e3b-517c794979d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:14.023068 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1350/2694] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:34:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:34:14.033365 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeaa2d01-a99d-4d2c-9ad2-5d11bf8a0ea0 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:14.034811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-eeaa2d01-a99d-4d2c-9ad2-5d11bf8a0ea0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:14.095669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeaa2d01-a99d-4d2c-9ad2-5d11bf8a0ea0 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:14.099262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bc5368-f29e-4fcc-a8ea-426519e6e563 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:14.099866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bc5368-f29e-4fcc-a8ea-426519e6e563 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:14.100169 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bc5368-f29e-4fcc-a8ea-426519e6e563 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:14.100441 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bc5368-f29e-4fcc-a8ea-426519e6e563 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:14.110404 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bc5368-f29e-4fcc-a8ea-426519e6e563 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:14.111836 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bc5368-f29e-4fcc-a8ea-426519e6e563 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:14.123168 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1345/2695] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:34:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:34:14.559704 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be62555-02a2-4c60-9e3b-517c794979d6 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:14.561377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6be62555-02a2-4c60-9e3b-517c794979d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:14.575530 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be62555-02a2-4c60-9e3b-517c794979d6 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:14.580697 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36483bc-9c68-4ebc-a750-afb325bf93d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:14.581372 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36483bc-9c68-4ebc-a750-afb325bf93d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:14.581648 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36483bc-9c68-4ebc-a750-afb325bf93d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:14.581923 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36483bc-9c68-4ebc-a750-afb325bf93d2 admin admin] PATH_INFO: `/v3/projects/6ea5e68c0fd84fc9b4e02d6b4d391918` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:14.583344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36483bc-9c68-4ebc-a750-afb325bf93d2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ea5e68c0fd84fc9b4e02d6b4d391918)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:14.585276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36483bc-9c68-4ebc-a750-afb325bf93d2 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:14.697067 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f36483bc-9c68-4ebc-a750-afb325bf93d2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ea5e68c0fd84fc9b4e02d6b4d391918', 'request_id': 'req-f36483bc-9c68-4ebc-a750-afb325bf93d2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:14.768936 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f36483bc-9c68-4ebc-a750-afb325bf93d2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ea5e68c0fd84fc9b4e02d6b4d391918', 'request_id': 'req-f36483bc-9c68-4ebc-a750-afb325bf93d2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:14.771308 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:14.776165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f36483bc-9c68-4ebc-a750-afb325bf93d2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ea5e68c0fd84fc9b4e02d6b4d391918', 'request_id': 'req-f36483bc-9c68-4ebc-a750-afb325bf93d2'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:14.788325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f36483bc-9c68-4ebc-a750-afb325bf93d2 admin admin] The token cache is being invalidate because project 6ea5e68c0fd84fc9b4e02d6b4d391918 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:34:14.788611 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f36483bc-9c68-4ebc-a750-afb325bf93d2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:14.789290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-f36483bc-9c68-4ebc-a750-afb325bf93d2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:14.790799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1351/2696] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:34:14 2022] DELETE /identity/v3/projects/6ea5e68c0fd84fc9b4e02d6b4d391918 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:14.795259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bc5368-f29e-4fcc-a8ea-426519e6e563 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:14.811332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bc5368-f29e-4fcc-a8ea-426519e6e563 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:14.863874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:14.892096 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefcc0e6-91ee-42c1-9775-13539f12d5b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:14.892762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefcc0e6-91ee-42c1-9775-13539f12d5b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:14.893025 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefcc0e6-91ee-42c1-9775-13539f12d5b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:14.893298 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefcc0e6-91ee-42c1-9775-13539f12d5b0 admin admin] PATH_INFO: `/v3/users/d8231a41df9b4646bda7fa2ef022715c` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:14.894498 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefcc0e6-91ee-42c1-9775-13539f12d5b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8231a41df9b4646bda7fa2ef022715c)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:14.896447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefcc0e6-91ee-42c1-9775-13539f12d5b0 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:15.105337 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-eefcc0e6-91ee-42c1-9775-13539f12d5b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8231a41df9b4646bda7fa2ef022715c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:15.114003 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-eefcc0e6-91ee-42c1-9775-13539f12d5b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8231a41df9b4646bda7fa2ef022715c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:15.114748 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-eefcc0e6-91ee-42c1-9775-13539f12d5b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8231a41df9b4646bda7fa2ef022715c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:15.128473 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-eefcc0e6-91ee-42c1-9775-13539f12d5b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8231a41df9b4646bda7fa2ef022715c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:15.187204 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-eefcc0e6-91ee-42c1-9775-13539f12d5b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8231a41df9b4646bda7fa2ef022715c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:15.197635 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1346/2697] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:34:14 2022] DELETE /identity/v3/users/d8231a41df9b4646bda7fa2ef022715c => generated 0 bytes in 403 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:34:15.684197 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36483bc-9c68-4ebc-a750-afb325bf93d2 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:15.685731 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-f36483bc-9c68-4ebc-a750-afb325bf93d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:15.703055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36483bc-9c68-4ebc-a750-afb325bf93d2 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:15.776706 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac704ab-3045-4d65-9193-d4f3a1bf986c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:15.777233 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac704ab-3045-4d65-9193-d4f3a1bf986c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:15.777516 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac704ab-3045-4d65-9193-d4f3a1bf986c admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:15.777789 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac704ab-3045-4d65-9193-d4f3a1bf986c admin admin] PATH_INFO: `/v3/projects/91082b44fa08489dab1cc89e3cff5a9e` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:15.779026 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac704ab-3045-4d65-9193-d4f3a1bf986c admin admin] RBAC: Authorizing `identity:delete_project(project_id=91082b44fa08489dab1cc89e3cff5a9e)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:15.781134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac704ab-3045-4d65-9193-d4f3a1bf986c admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:15.947281 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-6ac704ab-3045-4d65-9193-d4f3a1bf986c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91082b44fa08489dab1cc89e3cff5a9e', 'request_id': 'req-6ac704ab-3045-4d65-9193-d4f3a1bf986c'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:16.008638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-6ac704ab-3045-4d65-9193-d4f3a1bf986c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91082b44fa08489dab1cc89e3cff5a9e', 'request_id': 'req-6ac704ab-3045-4d65-9193-d4f3a1bf986c'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:16.014919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-6ac704ab-3045-4d65-9193-d4f3a1bf986c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91082b44fa08489dab1cc89e3cff5a9e', 'request_id': 'req-6ac704ab-3045-4d65-9193-d4f3a1bf986c'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:16.026095 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-6ac704ab-3045-4d65-9193-d4f3a1bf986c admin admin] The token cache is being invalidate because project 91082b44fa08489dab1cc89e3cff5a9e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:34:16.026378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-6ac704ab-3045-4d65-9193-d4f3a1bf986c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:16.027074 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-6ac704ab-3045-4d65-9193-d4f3a1bf986c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:16.028547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1352/2698] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:34:15 2022] DELETE /identity/v3/projects/91082b44fa08489dab1cc89e3cff5a9e => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:16.032542 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefcc0e6-91ee-42c1-9775-13539f12d5b0 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:16.033972 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-eefcc0e6-91ee-42c1-9775-13539f12d5b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:16.047224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefcc0e6-91ee-42c1-9775-13539f12d5b0 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:16.108576 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b820a919-23de-4ad0-aea9-a1588a78a0be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:16.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b820a919-23de-4ad0-aea9-a1588a78a0be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:16.109478 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b820a919-23de-4ad0-aea9-a1588a78a0be admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:16.109747 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b820a919-23de-4ad0-aea9-a1588a78a0be admin admin] PATH_INFO: `/v3/users/a790ec68cd674211b394905a98765940` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:16.110891 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b820a919-23de-4ad0-aea9-a1588a78a0be admin admin] RBAC: Authorizing `identity:delete_user(user_id=a790ec68cd674211b394905a98765940)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:16.112828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b820a919-23de-4ad0-aea9-a1588a78a0be admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:16.319808 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-b820a919-23de-4ad0-aea9-a1588a78a0be admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a790ec68cd674211b394905a98765940'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:16.324392 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:16.327445 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-b820a919-23de-4ad0-aea9-a1588a78a0be admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a790ec68cd674211b394905a98765940'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:16.328183 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-b820a919-23de-4ad0-aea9-a1588a78a0be admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a790ec68cd674211b394905a98765940'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:16.341419 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-b820a919-23de-4ad0-aea9-a1588a78a0be admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a790ec68cd674211b394905a98765940'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:16.418863 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-b820a919-23de-4ad0-aea9-a1588a78a0be admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a790ec68cd674211b394905a98765940'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:16.425646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1347/2699] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:34:16 2022] DELETE /identity/v3/users/a790ec68cd674211b394905a98765940 => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:16.862614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac704ab-3045-4d65-9193-d4f3a1bf986c admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:16.864122 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6ac704ab-3045-4d65-9193-d4f3a1bf986c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:16.878394 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac704ab-3045-4d65-9193-d4f3a1bf986c admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:16.938373 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8629d537-3640-471b-82e8-26c3eac263c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:16.939047 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8629d537-3640-471b-82e8-26c3eac263c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:16.939320 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8629d537-3640-471b-82e8-26c3eac263c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:16.939600 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8629d537-3640-471b-82e8-26c3eac263c6 admin admin] PATH_INFO: `/v3/projects/ad0d434b186f49a98eaf787d403f37cd` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:16.940722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8629d537-3640-471b-82e8-26c3eac263c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ad0d434b186f49a98eaf787d403f37cd)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:16.942606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8629d537-3640-471b-82e8-26c3eac263c6 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:16.964813 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:17.048005 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8629d537-3640-471b-82e8-26c3eac263c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ad0d434b186f49a98eaf787d403f37cd', 'request_id': 'req-8629d537-3640-471b-82e8-26c3eac263c6'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:17.134815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8629d537-3640-471b-82e8-26c3eac263c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ad0d434b186f49a98eaf787d403f37cd', 'request_id': 'req-8629d537-3640-471b-82e8-26c3eac263c6'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:17.141111 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8629d537-3640-471b-82e8-26c3eac263c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ad0d434b186f49a98eaf787d403f37cd', 'request_id': 'req-8629d537-3640-471b-82e8-26c3eac263c6'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:17.152463 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8629d537-3640-471b-82e8-26c3eac263c6 admin admin] The token cache is being invalidate because project ad0d434b186f49a98eaf787d403f37cd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:34:17.152751 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8629d537-3640-471b-82e8-26c3eac263c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:17.153409 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8629d537-3640-471b-82e8-26c3eac263c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:34:17.154925 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1353/2700] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:34:16 2022] DELETE /identity/v3/projects/ad0d434b186f49a98eaf787d403f37cd => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:17.159892 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5ad8a6-466a-4fac-85e9-79c5daf67980 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:17.160401 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5ad8a6-466a-4fac-85e9-79c5daf67980 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:17.160534 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5ad8a6-466a-4fac-85e9-79c5daf67980 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:17.181004 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-0b5ad8a6-466a-4fac-85e9-79c5daf67980 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:34:17.183269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-0b5ad8a6-466a-4fac-85e9-79c5daf67980 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:17.243145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1348/2701] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:34:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:34:17.251199 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c566d254-776c-481c-87fd-3c6387b64db2 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:17.251481 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c566d254-776c-481c-87fd-3c6387b64db2 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:17.251754 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c566d254-776c-481c-87fd-3c6387b64db2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:17.272061 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-c566d254-776c-481c-87fd-3c6387b64db2 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:34:17.273290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c566d254-776c-481c-87fd-3c6387b64db2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:17.281368 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1354/2702] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:34:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:34:17.285557 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5ad8a6-466a-4fac-85e9-79c5daf67980 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:17.295338 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5ad8a6-466a-4fac-85e9-79c5daf67980 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:17.300163 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4fef37-adcf-4155-aaf9-b47d13a24c9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:17.300756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4fef37-adcf-4155-aaf9-b47d13a24c9b admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:17.301029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4fef37-adcf-4155-aaf9-b47d13a24c9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:17.301297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4fef37-adcf-4155-aaf9-b47d13a24c9b admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:17.301925 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4fef37-adcf-4155-aaf9-b47d13a24c9b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:34:17.302220 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4fef37-adcf-4155-aaf9-b47d13a24c9b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:17.304394 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4fef37-adcf-4155-aaf9-b47d13a24c9b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:17.317000 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1349/2703] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:34:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:34:17.321877 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c566d254-776c-481c-87fd-3c6387b64db2 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:17.331322 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c566d254-776c-481c-87fd-3c6387b64db2 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:17.336752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f679a143-e9d8-4603-91eb-ec2fcb37a7b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:17.337343 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f679a143-e9d8-4603-91eb-ec2fcb37a7b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:17.337616 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f679a143-e9d8-4603-91eb-ec2fcb37a7b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:17.337879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f679a143-e9d8-4603-91eb-ec2fcb37a7b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:17.338525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f679a143-e9d8-4603-91eb-ec2fcb37a7b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:17.340436 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f679a143-e9d8-4603-91eb-ec2fcb37a7b5 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:17.451344 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1355/2704] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:34:17 2022] POST /identity/v3/projects => generated 379 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:34:17.455561 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4fef37-adcf-4155-aaf9-b47d13a24c9b admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:17.466373 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4fef37-adcf-4155-aaf9-b47d13a24c9b admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:17.525525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f46e6f-ba9e-4392-adc8-7f3b59909beb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:17.526133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f46e6f-ba9e-4392-adc8-7f3b59909beb admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:17.526414 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f46e6f-ba9e-4392-adc8-7f3b59909beb admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:17.526688 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f46e6f-ba9e-4392-adc8-7f3b59909beb admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:17.527317 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f46e6f-ba9e-4392-adc8-7f3b59909beb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:17.529257 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f46e6f-ba9e-4392-adc8-7f3b59909beb admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:17.659180 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1350/2705] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:34:17 2022] POST /identity/v3/users => generated 290 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:34:17.664292 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f679a143-e9d8-4603-91eb-ec2fcb37a7b5 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:17.674661 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f679a143-e9d8-4603-91eb-ec2fcb37a7b5 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:17.679850 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8b24a9-b21f-44bf-83e8-2438d4e14171 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:17.680491 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8b24a9-b21f-44bf-83e8-2438d4e14171 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:17.680761 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8b24a9-b21f-44bf-83e8-2438d4e14171 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:17.681027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8b24a9-b21f-44bf-83e8-2438d4e14171 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:17.681771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8b24a9-b21f-44bf-83e8-2438d4e14171 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:17.683350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8b24a9-b21f-44bf-83e8-2438d4e14171 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:17.693610 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1356/2706] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:34:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:34:17.697702 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f46e6f-ba9e-4392-adc8-7f3b59909beb admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:17.708535 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f46e6f-ba9e-4392-adc8-7f3b59909beb admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:17.713509 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d429971-c757-4b6e-ab51-2fa66596936c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:17.714756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d429971-c757-4b6e-ab51-2fa66596936c admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:17.715044 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d429971-c757-4b6e-ab51-2fa66596936c admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:17.715360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d429971-c757-4b6e-ab51-2fa66596936c admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:17.715978 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d429971-c757-4b6e-ab51-2fa66596936c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:17.717626 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d429971-c757-4b6e-ab51-2fa66596936c admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:17.728695 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1351/2707] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:34:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:34:17.734512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8b24a9-b21f-44bf-83e8-2438d4e14171 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:17.745416 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8b24a9-b21f-44bf-83e8-2438d4e14171 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:17.750147 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3767e3-1378-4542-b443-bb7c15cf2c99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:17.750797 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3767e3-1378-4542-b443-bb7c15cf2c99 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:17.751075 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3767e3-1378-4542-b443-bb7c15cf2c99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:17.751350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3767e3-1378-4542-b443-bb7c15cf2c99 admin admin] PATH_INFO: `/v3/projects/4124766f4f2843229059dbbdf7a2de74/users/fef6461d7ca14cab829412ce3b53852a/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:17.767370 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3767e3-1378-4542-b443-bb7c15cf2c99 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4124766f4f2843229059dbbdf7a2de74, user_id=fef6461d7ca14cab829412ce3b53852a, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:17.770116 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3767e3-1378-4542-b443-bb7c15cf2c99 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:17.849337 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1357/2708] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:34:17 2022] PUT /identity/v3/projects/4124766f4f2843229059dbbdf7a2de74/users/fef6461d7ca14cab829412ce3b53852a/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:17.855295 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4742b2-2f43-41e9-a61d-e38573a80c8d None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:17.855646 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4742b2-2f43-41e9-a61d-e38573a80c8d None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:17.855902 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4742b2-2f43-41e9-a61d-e38573a80c8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:17.874981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-5c4742b2-2f43-41e9-a61d-e38573a80c8d None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:34:17.876244 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5c4742b2-2f43-41e9-a61d-e38573a80c8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:17.907496 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:17.937209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1352/2709] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:34:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:34:17.950379 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3767e3-1378-4542-b443-bb7c15cf2c99 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:17.952946 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-5e3767e3-1378-4542-b443-bb7c15cf2c99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:18.014356 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3767e3-1378-4542-b443-bb7c15cf2c99 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:18.018044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfa4a06-51bf-485c-9018-f48f1caf1443 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:18.018710 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfa4a06-51bf-485c-9018-f48f1caf1443 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:18.018987 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfa4a06-51bf-485c-9018-f48f1caf1443 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:18.019257 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfa4a06-51bf-485c-9018-f48f1caf1443 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:18.028886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfa4a06-51bf-485c-9018-f48f1caf1443 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:18.030509 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfa4a06-51bf-485c-9018-f48f1caf1443 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:18.040830 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1358/2710] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:34:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:34:23.711386 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e992d0c7-439d-4ca6-bb16-09904f498174 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:23.711962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e992d0c7-439d-4ca6-bb16-09904f498174 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:23.712290 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e992d0c7-439d-4ca6-bb16-09904f498174 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:23.744946 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-e992d0c7-439d-4ca6-bb16-09904f498174 None None] MFA Rules not processed for user `fef6461d7ca14cab829412ce3b53852a`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:34:23.746042 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e992d0c7-439d-4ca6-bb16-09904f498174 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:23.837383 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1353/2711] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:34:23 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:34:23.842479 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfa4a06-51bf-485c-9018-f48f1caf1443 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:23.858896 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfa4a06-51bf-485c-9018-f48f1caf1443 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:23.863509 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f437f6-3e65-4c59-90bb-34092a38624e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:23.864151 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f437f6-3e65-4c59-90bb-34092a38624e admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:23.864434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f437f6-3e65-4c59-90bb-34092a38624e admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:23.864703 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f437f6-3e65-4c59-90bb-34092a38624e admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:23.865335 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f437f6-3e65-4c59-90bb-34092a38624e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:23.867266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f437f6-3e65-4c59-90bb-34092a38624e admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:23.977327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1359/2712] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:34:23 2022] POST /identity/v3/projects => generated 377 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:34:23.981674 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e992d0c7-439d-4ca6-bb16-09904f498174 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:23.993778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e992d0c7-439d-4ca6-bb16-09904f498174 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:24.055476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab2231a-daff-4e1e-9663-f15aa216230e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:24.055995 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab2231a-daff-4e1e-9663-f15aa216230e admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:24.056305 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab2231a-daff-4e1e-9663-f15aa216230e admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:24.056571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab2231a-daff-4e1e-9663-f15aa216230e admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:24.057189 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab2231a-daff-4e1e-9663-f15aa216230e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:24.059221 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab2231a-daff-4e1e-9663-f15aa216230e admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:24.158262 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1354/2713] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:34:23 2022] POST /identity/v3/users => generated 295 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:34:24.163280 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f437f6-3e65-4c59-90bb-34092a38624e admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:24.174330 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f437f6-3e65-4c59-90bb-34092a38624e admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:24.179151 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3229ee76-ad36-4ad2-b1fc-ad93f05aecb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:24.179765 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3229ee76-ad36-4ad2-b1fc-ad93f05aecb3 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:24.180041 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3229ee76-ad36-4ad2-b1fc-ad93f05aecb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:24.180351 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3229ee76-ad36-4ad2-b1fc-ad93f05aecb3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:24.183652 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3229ee76-ad36-4ad2-b1fc-ad93f05aecb3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:24.183652 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3229ee76-ad36-4ad2-b1fc-ad93f05aecb3 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:24.194618 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1360/2714] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:34:24 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:34:24.199596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab2231a-daff-4e1e-9663-f15aa216230e admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:24.220793 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab2231a-daff-4e1e-9663-f15aa216230e admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:24.232869 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c835c2a4-b006-4629-b0db-6caaa2900b18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:24.233549 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c835c2a4-b006-4629-b0db-6caaa2900b18 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:24.233883 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c835c2a4-b006-4629-b0db-6caaa2900b18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:24.234162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c835c2a4-b006-4629-b0db-6caaa2900b18 admin admin] PATH_INFO: `/v3/projects/fc202f480d314bad96b99d8bcb31330f/users/03c22bba97e94049bf33fe365eeceb81/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:24.264465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c835c2a4-b006-4629-b0db-6caaa2900b18 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc202f480d314bad96b99d8bcb31330f, user_id=03c22bba97e94049bf33fe365eeceb81, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:24.267162 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c835c2a4-b006-4629-b0db-6caaa2900b18 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:24.369792 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1355/2715] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:34:24 2022] PUT /identity/v3/projects/fc202f480d314bad96b99d8bcb31330f/users/03c22bba97e94049bf33fe365eeceb81/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:24.373958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3229ee76-ad36-4ad2-b1fc-ad93f05aecb3 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:24.384812 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3229ee76-ad36-4ad2-b1fc-ad93f05aecb3 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:24.393784 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:24.443442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2050bcd3-032a-49a2-ac2c-c0badd61fdaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:24.444835 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2050bcd3-032a-49a2-ac2c-c0badd61fdaa admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:24.445294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2050bcd3-032a-49a2-ac2c-c0badd61fdaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:24.445752 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2050bcd3-032a-49a2-ac2c-c0badd61fdaa admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:24.446854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2050bcd3-032a-49a2-ac2c-c0badd61fdaa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:24.449507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2050bcd3-032a-49a2-ac2c-c0badd61fdaa admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:24.468313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1361/2716] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:34:24 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:34:24.469841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c835c2a4-b006-4629-b0db-6caaa2900b18 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:24.485314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c835c2a4-b006-4629-b0db-6caaa2900b18 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:24.491279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0994fdf7-c3d4-4ea9-b059-d36925e48a85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:24.492217 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0994fdf7-c3d4-4ea9-b059-d36925e48a85 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:24.492493 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0994fdf7-c3d4-4ea9-b059-d36925e48a85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:24.492920 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0994fdf7-c3d4-4ea9-b059-d36925e48a85 admin admin] PATH_INFO: `/v3/projects/fc202f480d314bad96b99d8bcb31330f/users/03c22bba97e94049bf33fe365eeceb81/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:24.494909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0994fdf7-c3d4-4ea9-b059-d36925e48a85 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc202f480d314bad96b99d8bcb31330f, user_id=03c22bba97e94049bf33fe365eeceb81, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:24.497745 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0994fdf7-c3d4-4ea9-b059-d36925e48a85 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:24.515486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1356/2717] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:34:24 2022] PUT /identity/v3/projects/fc202f480d314bad96b99d8bcb31330f/users/03c22bba97e94049bf33fe365eeceb81/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:34:30.332631 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854b129e-e2ef-45f7-afeb-2dec870f1af3 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:30.334531 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854b129e-e2ef-45f7-afeb-2dec870f1af3 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:30.334531 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854b129e-e2ef-45f7-afeb-2dec870f1af3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:30.377290 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-854b129e-e2ef-45f7-afeb-2dec870f1af3 None None] MFA Rules not processed for user `03c22bba97e94049bf33fe365eeceb81`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:34:30.378638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-854b129e-e2ef-45f7-afeb-2dec870f1af3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:30.468073 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1362/2718] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:34:30 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:34:30.485498 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0994fdf7-c3d4-4ea9-b059-d36925e48a85 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:34:30.489473 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-0994fdf7-c3d4-4ea9-b059-d36925e48a85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:30.539224 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:30.628013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0994fdf7-c3d4-4ea9-b059-d36925e48a85 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:34:30.634515 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8db075-2317-441b-bcfa-a7dab21b0242 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:34:30.635134 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8db075-2317-441b-bcfa-a7dab21b0242 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:30.635443 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8db075-2317-441b-bcfa-a7dab21b0242 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:30.635674 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8db075-2317-441b-bcfa-a7dab21b0242 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:30.700731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8db075-2317-441b-bcfa-a7dab21b0242 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:34:30.702381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8db075-2317-441b-bcfa-a7dab21b0242 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:34:30.713389 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1357/2719] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:34:30 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:34:40.464452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=57 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:40.707603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=60 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:34:51.471009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c2556b-b62c-4ca5-8835-6368c5a257a7 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:34:51.471541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c2556b-b62c-4ca5-8835-6368c5a257a7 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:34:51.471541 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c2556b-b62c-4ca5-8835-6368c5a257a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:34:51.490716 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-54c2556b-b62c-4ca5-8835-6368c5a257a7 None None] MFA Rules not processed for user `fef6461d7ca14cab829412ce3b53852a`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:34:51.491922 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-54c2556b-b62c-4ca5-8835-6368c5a257a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:34:51.499356 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1363/2720] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:34:51 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:35:01.199171 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8db075-2317-441b-bcfa-a7dab21b0242 service nova] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:35:01.258520 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8db075-2317-441b-bcfa-a7dab21b0242 service nova] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:35:01.262933 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a3d927-1f3f-473a-af19-ebdf55126bad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:35:01.269371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a3d927-1f3f-473a-af19-ebdf55126bad service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:35:01.269371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a3d927-1f3f-473a-af19-ebdf55126bad service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:35:01.269371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a3d927-1f3f-473a-af19-ebdf55126bad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:35:01.269371 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-19a3d927-1f3f-473a-af19-ebdf55126bad service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:35:01.281213 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a3d927-1f3f-473a-af19-ebdf55126bad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:35:01.282648 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a3d927-1f3f-473a-af19-ebdf55126bad service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:35:01.294763 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1358/2721] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:35:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:35:01.485456 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:35:02.586858 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c2556b-b62c-4ca5-8835-6368c5a257a7 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:35:02.602629 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c2556b-b62c-4ca5-8835-6368c5a257a7 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:35:02.606926 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3304e7-cc47-4538-a5d5-0dac22008df1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:35:02.607674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3304e7-cc47-4538-a5d5-0dac22008df1 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:35:02.608093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3304e7-cc47-4538-a5d5-0dac22008df1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:35:02.608403 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3304e7-cc47-4538-a5d5-0dac22008df1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:35:02.611363 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9b3304e7-cc47-4538-a5d5-0dac22008df1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:35:02.628203 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3304e7-cc47-4538-a5d5-0dac22008df1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:35:02.632799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3304e7-cc47-4538-a5d5-0dac22008df1 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:35:02.644248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1364/2722] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:35:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:35:06.222777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5208e09c-7ea0-451c-9ff2-df4637b36836 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:35:06.222777 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5208e09c-7ea0-451c-9ff2-df4637b36836 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:35:06.226158 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5208e09c-7ea0-451c-9ff2-df4637b36836 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:35:06.245766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-5208e09c-7ea0-451c-9ff2-df4637b36836 None None] MFA Rules not processed for user `a91ac0ba7e8c4418930852b0fa184247`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:35:06.247157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5208e09c-7ea0-451c-9ff2-df4637b36836 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:35:06.316091 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1359/2723] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:35:06 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:35:12.638131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=6 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:35:13.991880 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3304e7-cc47-4538-a5d5-0dac22008df1 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:35:13.993484 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9b3304e7-cc47-4538-a5d5-0dac22008df1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:35:14.007942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3304e7-cc47-4538-a5d5-0dac22008df1 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:35:14.012057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162df4c9-367d-4f2b-b5c8-c352558e8a51 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:35:14.013240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162df4c9-367d-4f2b-b5c8-c352558e8a51 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:35:14.013418 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162df4c9-367d-4f2b-b5c8-c352558e8a51 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:35:14.013601 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162df4c9-367d-4f2b-b5c8-c352558e8a51 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:35:14.023438 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162df4c9-367d-4f2b-b5c8-c352558e8a51 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:35:14.024919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162df4c9-367d-4f2b-b5c8-c352558e8a51 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:35:14.035309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1365/2724] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:35:13 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:35:16.302897 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=51 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:35:24.030578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=6 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:36:08.842765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5208e09c-7ea0-451c-9ff2-df4637b36836 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:08.844415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5208e09c-7ea0-451c-9ff2-df4637b36836 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:36:08.901820 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5208e09c-7ea0-451c-9ff2-df4637b36836 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:08.905577 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22adcda-70da-458e-a3cc-a418ba9f1350 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:08.906233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22adcda-70da-458e-a3cc-a418ba9f1350 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:08.906512 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22adcda-70da-458e-a3cc-a418ba9f1350 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:08.906783 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22adcda-70da-458e-a3cc-a418ba9f1350 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:08.908824 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f22adcda-70da-458e-a3cc-a418ba9f1350 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:36:08.921193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22adcda-70da-458e-a3cc-a418ba9f1350 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:08.922634 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22adcda-70da-458e-a3cc-a418ba9f1350 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:08.932005 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1360/2725] 10.222.0.47 () {60 vars in 1396 bytes} [Sun Aug 21 01:36:08 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3260 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:36:14.697228 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162df4c9-367d-4f2b-b5c8-c352558e8a51 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:14.719367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-162df4c9-367d-4f2b-b5c8-c352558e8a51 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:36:14.719367 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162df4c9-367d-4f2b-b5c8-c352558e8a51 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:14.721037 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c1fee9-0320-4508-a8f4-0093b55383ca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:14.721650 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c1fee9-0320-4508-a8f4-0093b55383ca service nova] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:14.721924 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c1fee9-0320-4508-a8f4-0093b55383ca service nova] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:14.722189 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c1fee9-0320-4508-a8f4-0093b55383ca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:14.723962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c8c1fee9-0320-4508-a8f4-0093b55383ca service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:36:14.738836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c1fee9-0320-4508-a8f4-0093b55383ca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:14.740338 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c1fee9-0320-4508-a8f4-0093b55383ca service nova] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:14.753776 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1366/2726] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:36:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:36:17.430657 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22adcda-70da-458e-a3cc-a418ba9f1350 service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:17.440780 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22adcda-70da-458e-a3cc-a418ba9f1350 service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:17.444380 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdd47b3-4fac-485c-9f9d-46535ec18de7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:17.444967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdd47b3-4fac-485c-9f9d-46535ec18de7 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:17.445238 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdd47b3-4fac-485c-9f9d-46535ec18de7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:17.445540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdd47b3-4fac-485c-9f9d-46535ec18de7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:17.447324 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7fdd47b3-4fac-485c-9f9d-46535ec18de7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:36:17.458049 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdd47b3-4fac-485c-9f9d-46535ec18de7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:17.461901 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdd47b3-4fac-485c-9f9d-46535ec18de7 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:17.467852 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1361/2727] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:36:17 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:36:21.405873 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c1fee9-0320-4508-a8f4-0093b55383ca service nova] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:21.415936 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c1fee9-0320-4508-a8f4-0093b55383ca service nova] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:21.419587 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc96f4d-344f-432e-ac02-035fa4eb8f7c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:21.420242 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc96f4d-344f-432e-ac02-035fa4eb8f7c service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:21.420522 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc96f4d-344f-432e-ac02-035fa4eb8f7c service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:21.420791 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc96f4d-344f-432e-ac02-035fa4eb8f7c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:21.422603 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6cc96f4d-344f-432e-ac02-035fa4eb8f7c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:36:21.496092 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc96f4d-344f-432e-ac02-035fa4eb8f7c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:21.499015 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc96f4d-344f-432e-ac02-035fa4eb8f7c service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:21.509859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1367/2728] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:36:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:36:25.657569 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdd47b3-4fac-485c-9f9d-46535ec18de7 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:25.659165 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-7fdd47b3-4fac-485c-9f9d-46535ec18de7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:36:25.675522 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdd47b3-4fac-485c-9f9d-46535ec18de7 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:25.680263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3cb7e5-02ca-4dcd-bb5a-2d0a7c681bc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:25.680913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3cb7e5-02ca-4dcd-bb5a-2d0a7c681bc2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:25.681191 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3cb7e5-02ca-4dcd-bb5a-2d0a7c681bc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:25.681463 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3cb7e5-02ca-4dcd-bb5a-2d0a7c681bc2 admin admin] PATH_INFO: `/v3/users/a91ac0ba7e8c4418930852b0fa184247` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:25.682590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3cb7e5-02ca-4dcd-bb5a-2d0a7c681bc2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a91ac0ba7e8c4418930852b0fa184247)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:25.684533 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3cb7e5-02ca-4dcd-bb5a-2d0a7c681bc2 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:25.843557 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ea3cb7e5-02ca-4dcd-bb5a-2d0a7c681bc2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a91ac0ba7e8c4418930852b0fa184247'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:36:25.850335 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ea3cb7e5-02ca-4dcd-bb5a-2d0a7c681bc2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a91ac0ba7e8c4418930852b0fa184247'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:36:25.851032 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ea3cb7e5-02ca-4dcd-bb5a-2d0a7c681bc2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a91ac0ba7e8c4418930852b0fa184247'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:36:25.862631 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ea3cb7e5-02ca-4dcd-bb5a-2d0a7c681bc2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a91ac0ba7e8c4418930852b0fa184247'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:36:25.949656 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-ea3cb7e5-02ca-4dcd-bb5a-2d0a7c681bc2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a91ac0ba7e8c4418930852b0fa184247'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:36:25.956481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1362/2729] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:36:25 2022] DELETE /identity/v3/users/a91ac0ba7e8c4418930852b0fa184247 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:36:26.185316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc96f4d-344f-432e-ac02-035fa4eb8f7c service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:26.197145 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc96f4d-344f-432e-ac02-035fa4eb8f7c service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:26.271974 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2e72f3-1d6b-48d2-a00c-c1fa7c6bfb52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:26.272702 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2e72f3-1d6b-48d2-a00c-c1fa7c6bfb52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:26.276644 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2e72f3-1d6b-48d2-a00c-c1fa7c6bfb52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:26.276644 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2e72f3-1d6b-48d2-a00c-c1fa7c6bfb52 admin admin] PATH_INFO: `/v3/projects/122b46d9e29843a88c7ca5f019426539` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:26.276644 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2e72f3-1d6b-48d2-a00c-c1fa7c6bfb52 admin admin] RBAC: Authorizing `identity:delete_project(project_id=122b46d9e29843a88c7ca5f019426539)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:26.276644 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2e72f3-1d6b-48d2-a00c-c1fa7c6bfb52 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:26.402799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2a2e72f3-1d6b-48d2-a00c-c1fa7c6bfb52 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '122b46d9e29843a88c7ca5f019426539', 'request_id': 'req-2a2e72f3-1d6b-48d2-a00c-c1fa7c6bfb52'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:36:26.480855 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2a2e72f3-1d6b-48d2-a00c-c1fa7c6bfb52 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '122b46d9e29843a88c7ca5f019426539', 'request_id': 'req-2a2e72f3-1d6b-48d2-a00c-c1fa7c6bfb52'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:36:26.486984 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2a2e72f3-1d6b-48d2-a00c-c1fa7c6bfb52 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '122b46d9e29843a88c7ca5f019426539', 'request_id': 'req-2a2e72f3-1d6b-48d2-a00c-c1fa7c6bfb52'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:36:26.498871 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2a2e72f3-1d6b-48d2-a00c-c1fa7c6bfb52 admin admin] The token cache is being invalidate because project 122b46d9e29843a88c7ca5f019426539 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:36:26.499155 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2a2e72f3-1d6b-48d2-a00c-c1fa7c6bfb52 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:36:26.499782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2a2e72f3-1d6b-48d2-a00c-c1fa7c6bfb52 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:36:26.501325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1368/2730] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:36:26 2022] DELETE /identity/v3/projects/122b46d9e29843a88c7ca5f019426539 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:36:26.506439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7171f19-5f3f-4d3a-8263-1647a79a0473 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:26.507623 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7171f19-5f3f-4d3a-8263-1647a79a0473 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:26.507623 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7171f19-5f3f-4d3a-8263-1647a79a0473 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:26.526798 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-c7171f19-5f3f-4d3a-8263-1647a79a0473 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:36:26.527996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-c7171f19-5f3f-4d3a-8263-1647a79a0473 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:36:26.591067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1363/2731] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:36:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:36:26.599104 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478d6772-2c83-4187-ab9f-ff8c125d029c None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:26.599381 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478d6772-2c83-4187-ab9f-ff8c125d029c None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:26.599652 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478d6772-2c83-4187-ab9f-ff8c125d029c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:26.619918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-478d6772-2c83-4187-ab9f-ff8c125d029c None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:36:26.621198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-478d6772-2c83-4187-ab9f-ff8c125d029c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:36:26.630296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1369/2732] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:36:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:36:26.634438 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7171f19-5f3f-4d3a-8263-1647a79a0473 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:26.639259 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:36:26.644538 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7171f19-5f3f-4d3a-8263-1647a79a0473 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:26.649637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2829a67e-7849-4109-bdef-f8b99cfb4528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:26.654087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2829a67e-7849-4109-bdef-f8b99cfb4528 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:26.654087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2829a67e-7849-4109-bdef-f8b99cfb4528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:26.654087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2829a67e-7849-4109-bdef-f8b99cfb4528 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:26.654087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2829a67e-7849-4109-bdef-f8b99cfb4528 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:36:26.654087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2829a67e-7849-4109-bdef-f8b99cfb4528 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:26.654087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2829a67e-7849-4109-bdef-f8b99cfb4528 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:26.666168 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1364/2733] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:36:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:36:26.671063 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478d6772-2c83-4187-ab9f-ff8c125d029c None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:26.676914 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:36:26.688499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478d6772-2c83-4187-ab9f-ff8c125d029c None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:26.688499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07889382-00ee-4c5b-8091-4a67568539f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:26.688499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07889382-00ee-4c5b-8091-4a67568539f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:26.688499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07889382-00ee-4c5b-8091-4a67568539f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:26.688941 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07889382-00ee-4c5b-8091-4a67568539f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:26.688941 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07889382-00ee-4c5b-8091-4a67568539f7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:26.689780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07889382-00ee-4c5b-8091-4a67568539f7 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:26.814385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1370/2734] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:36:26 2022] POST /identity/v3/projects => generated 371 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:36:26.818743 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2829a67e-7849-4109-bdef-f8b99cfb4528 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:26.829056 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2829a67e-7849-4109-bdef-f8b99cfb4528 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:26.886158 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d96441-8de1-4552-ac4c-8bd918e8a61b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:26.886874 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d96441-8de1-4552-ac4c-8bd918e8a61b admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:26.887622 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d96441-8de1-4552-ac4c-8bd918e8a61b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:26.887898 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d96441-8de1-4552-ac4c-8bd918e8a61b admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:26.888562 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d96441-8de1-4552-ac4c-8bd918e8a61b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:26.890485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d96441-8de1-4552-ac4c-8bd918e8a61b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:27.031093 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1365/2735] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:36:26 2022] POST /identity/v3/users => generated 286 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:36:27.035434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07889382-00ee-4c5b-8091-4a67568539f7 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:27.046406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07889382-00ee-4c5b-8091-4a67568539f7 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:27.050934 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7f8acb-8469-494c-8e87-42677627ca9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:27.051528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7f8acb-8469-494c-8e87-42677627ca9d admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:27.051803 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7f8acb-8469-494c-8e87-42677627ca9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:27.052071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7f8acb-8469-494c-8e87-42677627ca9d admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:27.052764 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7f8acb-8469-494c-8e87-42677627ca9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:27.054350 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7f8acb-8469-494c-8e87-42677627ca9d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:27.064355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1371/2736] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:36:27 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:36:27.068711 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d96441-8de1-4552-ac4c-8bd918e8a61b admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:27.079096 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d96441-8de1-4552-ac4c-8bd918e8a61b admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:27.083959 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c519c14-5aaa-4196-964b-3b8d09117a6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:27.084841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c519c14-5aaa-4196-964b-3b8d09117a6c admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:27.085118 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c519c14-5aaa-4196-964b-3b8d09117a6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:27.085412 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c519c14-5aaa-4196-964b-3b8d09117a6c admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:27.086039 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c519c14-5aaa-4196-964b-3b8d09117a6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:27.087613 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c519c14-5aaa-4196-964b-3b8d09117a6c admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:27.098716 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1366/2737] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:36:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:36:27.102879 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7f8acb-8469-494c-8e87-42677627ca9d admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:27.114293 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7f8acb-8469-494c-8e87-42677627ca9d admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:27.118912 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ad1041-e059-445a-a86a-25d9191f8d46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:27.119566 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ad1041-e059-445a-a86a-25d9191f8d46 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:27.119838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ad1041-e059-445a-a86a-25d9191f8d46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:27.120125 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ad1041-e059-445a-a86a-25d9191f8d46 admin admin] PATH_INFO: `/v3/projects/bf1c228400194d4ab229bb4ff0cfae52/users/2ea1e3053c674d45a1f4d0bd419604a9/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:27.134828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ad1041-e059-445a-a86a-25d9191f8d46 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bf1c228400194d4ab229bb4ff0cfae52, user_id=2ea1e3053c674d45a1f4d0bd419604a9, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:27.137523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ad1041-e059-445a-a86a-25d9191f8d46 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:27.247294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1372/2738] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:36:27 2022] PUT /identity/v3/projects/bf1c228400194d4ab229bb4ff0cfae52/users/2ea1e3053c674d45a1f4d0bd419604a9/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:36:27.258609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4138771e-91ee-47d9-a449-ffbe38254e69 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:27.258891 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4138771e-91ee-47d9-a449-ffbe38254e69 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:27.259192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4138771e-91ee-47d9-a449-ffbe38254e69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:27.287254 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-4138771e-91ee-47d9-a449-ffbe38254e69 None None] MFA Rules not processed for user `2ea1e3053c674d45a1f4d0bd419604a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:36:27.288518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-4138771e-91ee-47d9-a449-ffbe38254e69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:36:27.355628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1367/2739] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:36:27 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:36:27.360115 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ad1041-e059-445a-a86a-25d9191f8d46 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:27.423798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ad1041-e059-445a-a86a-25d9191f8d46 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:27.428955 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c56094a-941f-430e-be58-44f0835d9ec6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:27.429950 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c56094a-941f-430e-be58-44f0835d9ec6 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:27.429950 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c56094a-941f-430e-be58-44f0835d9ec6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:27.430670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c56094a-941f-430e-be58-44f0835d9ec6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:27.431130 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c56094a-941f-430e-be58-44f0835d9ec6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:27.433075 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c56094a-941f-430e-be58-44f0835d9ec6 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:27.539981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1373/2740] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:36:27 2022] POST /identity/v3/projects => generated 373 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:36:27.544227 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4138771e-91ee-47d9-a449-ffbe38254e69 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:27.555074 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4138771e-91ee-47d9-a449-ffbe38254e69 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:27.611088 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:36:27.618203 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad50cc5-8c2d-489c-b00a-ca30fa1cbb88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:27.628400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad50cc5-8c2d-489c-b00a-ca30fa1cbb88 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:27.628400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad50cc5-8c2d-489c-b00a-ca30fa1cbb88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:27.628400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad50cc5-8c2d-489c-b00a-ca30fa1cbb88 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:27.628400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad50cc5-8c2d-489c-b00a-ca30fa1cbb88 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:27.628400 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad50cc5-8c2d-489c-b00a-ca30fa1cbb88 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:27.727628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1368/2741] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:36:27 2022] POST /identity/v3/users => generated 293 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:36:27.731711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c56094a-941f-430e-be58-44f0835d9ec6 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:27.742566 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c56094a-941f-430e-be58-44f0835d9ec6 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:27.749923 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e68ebd1-b625-4056-8c17-01d17029dd18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:27.750874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e68ebd1-b625-4056-8c17-01d17029dd18 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:27.751310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e68ebd1-b625-4056-8c17-01d17029dd18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:27.751780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e68ebd1-b625-4056-8c17-01d17029dd18 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:27.752557 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e68ebd1-b625-4056-8c17-01d17029dd18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:27.754214 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e68ebd1-b625-4056-8c17-01d17029dd18 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:27.766288 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1374/2742] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:36:27 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:36:27.770578 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad50cc5-8c2d-489c-b00a-ca30fa1cbb88 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:27.783333 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad50cc5-8c2d-489c-b00a-ca30fa1cbb88 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:27.787973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462509a3-9f2a-4a68-a144-5587216d26ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:27.788665 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462509a3-9f2a-4a68-a144-5587216d26ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:27.788940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462509a3-9f2a-4a68-a144-5587216d26ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:27.789212 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462509a3-9f2a-4a68-a144-5587216d26ed admin admin] PATH_INFO: `/v3/projects/863e6ce5f84842e98cdda3dfdb11f0c1/users/8972cec60e10486d9f0c3d105c20de29/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:27.803732 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462509a3-9f2a-4a68-a144-5587216d26ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=863e6ce5f84842e98cdda3dfdb11f0c1, user_id=8972cec60e10486d9f0c3d105c20de29, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:27.806376 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462509a3-9f2a-4a68-a144-5587216d26ed admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:27.887367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1369/2743] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:36:27 2022] PUT /identity/v3/projects/863e6ce5f84842e98cdda3dfdb11f0c1/users/8972cec60e10486d9f0c3d105c20de29/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:36:27.892196 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e68ebd1-b625-4056-8c17-01d17029dd18 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:27.902615 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e68ebd1-b625-4056-8c17-01d17029dd18 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:27.966850 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931c4658-e4c4-4cc3-bd89-4855ac586511 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:27.967461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931c4658-e4c4-4cc3-bd89-4855ac586511 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:27.967759 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931c4658-e4c4-4cc3-bd89-4855ac586511 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:27.968014 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931c4658-e4c4-4cc3-bd89-4855ac586511 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:27.968696 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931c4658-e4c4-4cc3-bd89-4855ac586511 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:27.970301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931c4658-e4c4-4cc3-bd89-4855ac586511 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:27.980845 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1375/2744] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:36:27 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:36:27.985774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462509a3-9f2a-4a68-a144-5587216d26ed admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:27.996360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462509a3-9f2a-4a68-a144-5587216d26ed admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:28.001596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac4e347-6d0c-4d01-abbd-fd278e2d9a6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:28.002244 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac4e347-6d0c-4d01-abbd-fd278e2d9a6e admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:28.002517 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac4e347-6d0c-4d01-abbd-fd278e2d9a6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:28.002784 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac4e347-6d0c-4d01-abbd-fd278e2d9a6e admin admin] PATH_INFO: `/v3/projects/863e6ce5f84842e98cdda3dfdb11f0c1/users/8972cec60e10486d9f0c3d105c20de29/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:28.004773 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac4e347-6d0c-4d01-abbd-fd278e2d9a6e admin admin] RBAC: Authorizing `identity:create_grant(project_id=863e6ce5f84842e98cdda3dfdb11f0c1, user_id=8972cec60e10486d9f0c3d105c20de29, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:28.007439 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac4e347-6d0c-4d01-abbd-fd278e2d9a6e admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:28.019665 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1370/2745] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:36:27 2022] PUT /identity/v3/projects/863e6ce5f84842e98cdda3dfdb11f0c1/users/8972cec60e10486d9f0c3d105c20de29/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:36:28.035973 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d242fb-33ec-4e52-8a4a-1f3cb6deeef1 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:28.036317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d242fb-33ec-4e52-8a4a-1f3cb6deeef1 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:28.036588 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d242fb-33ec-4e52-8a4a-1f3cb6deeef1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:28.069507 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-b9d242fb-33ec-4e52-8a4a-1f3cb6deeef1 None None] MFA Rules not processed for user `8972cec60e10486d9f0c3d105c20de29`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:36:28.070730 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-b9d242fb-33ec-4e52-8a4a-1f3cb6deeef1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:36:28.077134 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:36:28.151151 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1376/2746] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:36:28 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:36:28.162133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac4e347-6d0c-4d01-abbd-fd278e2d9a6e admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:28.163955 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5ac4e347-6d0c-4d01-abbd-fd278e2d9a6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:36:28.236454 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac4e347-6d0c-4d01-abbd-fd278e2d9a6e admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:28.240997 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166eb272-289f-47ba-b05d-5e4539c8071c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:28.241602 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166eb272-289f-47ba-b05d-5e4539c8071c service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:28.241875 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166eb272-289f-47ba-b05d-5e4539c8071c service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:28.242143 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166eb272-289f-47ba-b05d-5e4539c8071c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:28.251933 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166eb272-289f-47ba-b05d-5e4539c8071c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:28.253391 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166eb272-289f-47ba-b05d-5e4539c8071c service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:28.262929 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1371/2747] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:36:28 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:36:28.485160 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d242fb-33ec-4e52-8a4a-1f3cb6deeef1 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:36:28.495044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d242fb-33ec-4e52-8a4a-1f3cb6deeef1 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:36:28.498627 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9150d27-81a8-4a9c-a88e-a2965f2fa6b3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:36:28.499238 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9150d27-81a8-4a9c-a88e-a2965f2fa6b3 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:36:28.499513 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9150d27-81a8-4a9c-a88e-a2965f2fa6b3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:36:28.499784 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9150d27-81a8-4a9c-a88e-a2965f2fa6b3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:36:28.561325 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9150d27-81a8-4a9c-a88e-a2965f2fa6b3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:36:28.562828 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9150d27-81a8-4a9c-a88e-a2965f2fa6b3 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:36:28.572711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1377/2748] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:36:28 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:36:38.258637 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=44,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:36:38.567990 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=54 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:17.839410 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166eb272-289f-47ba-b05d-5e4539c8071c service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:17.906593 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166eb272-289f-47ba-b05d-5e4539c8071c service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:17.911634 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c99a92b-90bd-4430-874e-3240f36286ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:17.912302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c99a92b-90bd-4430-874e-3240f36286ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:17.912570 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c99a92b-90bd-4430-874e-3240f36286ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:17.912838 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c99a92b-90bd-4430-874e-3240f36286ff admin admin] PATH_INFO: `/v3/users/2ea1e3053c674d45a1f4d0bd419604a9` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:17.913948 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c99a92b-90bd-4430-874e-3240f36286ff admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ea1e3053c674d45a1f4d0bd419604a9)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:17.915865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c99a92b-90bd-4430-874e-3240f36286ff admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:18.071692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1c99a92b-90bd-4430-874e-3240f36286ff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ea1e3053c674d45a1f4d0bd419604a9'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:18.078897 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1c99a92b-90bd-4430-874e-3240f36286ff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ea1e3053c674d45a1f4d0bd419604a9'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:18.079565 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1c99a92b-90bd-4430-874e-3240f36286ff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ea1e3053c674d45a1f4d0bd419604a9'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:18.092968 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1c99a92b-90bd-4430-874e-3240f36286ff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ea1e3053c674d45a1f4d0bd419604a9'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:18.165384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-1c99a92b-90bd-4430-874e-3240f36286ff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ea1e3053c674d45a1f4d0bd419604a9'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:18.172544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1372/2749] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:37:17 2022] DELETE /identity/v3/users/2ea1e3053c674d45a1f4d0bd419604a9 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:18.177310 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dca0e5-7d6e-48a2-a1dc-6bb2f2bf1eef None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:18.177809 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dca0e5-7d6e-48a2-a1dc-6bb2f2bf1eef None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:18.177930 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dca0e5-7d6e-48a2-a1dc-6bb2f2bf1eef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:18.198100 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-e5dca0e5-7d6e-48a2-a1dc-6bb2f2bf1eef None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:18.201340 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e5dca0e5-7d6e-48a2-a1dc-6bb2f2bf1eef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:18.385265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1378/2750] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:18.394991 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c99a92b-90bd-4430-874e-3240f36286ff admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:18.396994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-1c99a92b-90bd-4430-874e-3240f36286ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:18.453598 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c99a92b-90bd-4430-874e-3240f36286ff admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:18.457240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed237b0-5f99-49e8-a1c9-52b1d7c9e138 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:18.458361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed237b0-5f99-49e8-a1c9-52b1d7c9e138 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:18.458638 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed237b0-5f99-49e8-a1c9-52b1d7c9e138 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:18.458906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed237b0-5f99-49e8-a1c9-52b1d7c9e138 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:18.468208 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed237b0-5f99-49e8-a1c9-52b1d7c9e138 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:18.469627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed237b0-5f99-49e8-a1c9-52b1d7c9e138 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:18.479723 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1373/2751] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:37:18.917943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dca0e5-7d6e-48a2-a1dc-6bb2f2bf1eef None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:18.917943 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e5dca0e5-7d6e-48a2-a1dc-6bb2f2bf1eef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:18.930073 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dca0e5-7d6e-48a2-a1dc-6bb2f2bf1eef None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:18.934758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e085ec59-3bba-4670-92c6-370eb75c4504 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:18.935428 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e085ec59-3bba-4670-92c6-370eb75c4504 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:18.935654 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e085ec59-3bba-4670-92c6-370eb75c4504 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:18.935919 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e085ec59-3bba-4670-92c6-370eb75c4504 admin admin] PATH_INFO: `/v3/projects/bf1c228400194d4ab229bb4ff0cfae52` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:18.937064 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e085ec59-3bba-4670-92c6-370eb75c4504 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bf1c228400194d4ab229bb4ff0cfae52)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:18.938965 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e085ec59-3bba-4670-92c6-370eb75c4504 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:19.069257 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e085ec59-3bba-4670-92c6-370eb75c4504 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bf1c228400194d4ab229bb4ff0cfae52', 'request_id': 'req-e085ec59-3bba-4670-92c6-370eb75c4504'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:19.142035 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e085ec59-3bba-4670-92c6-370eb75c4504 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bf1c228400194d4ab229bb4ff0cfae52', 'request_id': 'req-e085ec59-3bba-4670-92c6-370eb75c4504'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:19.147791 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e085ec59-3bba-4670-92c6-370eb75c4504 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bf1c228400194d4ab229bb4ff0cfae52', 'request_id': 'req-e085ec59-3bba-4670-92c6-370eb75c4504'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:19.159105 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e085ec59-3bba-4670-92c6-370eb75c4504 admin admin] The token cache is being invalidate because project bf1c228400194d4ab229bb4ff0cfae52 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:37:19.159384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e085ec59-3bba-4670-92c6-370eb75c4504 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:19.160012 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-e085ec59-3bba-4670-92c6-370eb75c4504 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:19.161496 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1379/2752] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:37:18 2022] DELETE /identity/v3/projects/bf1c228400194d4ab229bb4ff0cfae52 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:19.165451 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed237b0-5f99-49e8-a1c9-52b1d7c9e138 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:19.175164 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed237b0-5f99-49e8-a1c9-52b1d7c9e138 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:19.194668 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:19.232252 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef3f889-166a-49bc-9b16-4d46c7bde3f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:19.232879 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef3f889-166a-49bc-9b16-4d46c7bde3f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:19.233158 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef3f889-166a-49bc-9b16-4d46c7bde3f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:19.233428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef3f889-166a-49bc-9b16-4d46c7bde3f0 admin admin] PATH_INFO: `/v3/users/8972cec60e10486d9f0c3d105c20de29` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:19.234626 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef3f889-166a-49bc-9b16-4d46c7bde3f0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8972cec60e10486d9f0c3d105c20de29)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:19.236578 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef3f889-166a-49bc-9b16-4d46c7bde3f0 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:19.310281 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3371db-7770-45c3-af1b-36f3f87e2831 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:19.310741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3371db-7770-45c3-af1b-36f3f87e2831 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:19.310906 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3371db-7770-45c3-af1b-36f3f87e2831 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:19.329364 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-6a3371db-7770-45c3-af1b-36f3f87e2831 None None] MFA Rules not processed for user `fef6461d7ca14cab829412ce3b53852a`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:19.330595 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6a3371db-7770-45c3-af1b-36f3f87e2831 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:19.390317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1380/2753] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:19 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:37:19.419558 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6ef3f889-166a-49bc-9b16-4d46c7bde3f0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8972cec60e10486d9f0c3d105c20de29'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:19.428865 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6ef3f889-166a-49bc-9b16-4d46c7bde3f0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8972cec60e10486d9f0c3d105c20de29'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:19.429584 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6ef3f889-166a-49bc-9b16-4d46c7bde3f0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8972cec60e10486d9f0c3d105c20de29'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:19.442063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6ef3f889-166a-49bc-9b16-4d46c7bde3f0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8972cec60e10486d9f0c3d105c20de29'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:19.485495 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-6ef3f889-166a-49bc-9b16-4d46c7bde3f0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8972cec60e10486d9f0c3d105c20de29'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:19.492709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1374/2754] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:37:19 2022] DELETE /identity/v3/users/8972cec60e10486d9f0c3d105c20de29 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:19.713112 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3371db-7770-45c3-af1b-36f3f87e2831 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:19.714678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6a3371db-7770-45c3-af1b-36f3f87e2831 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:19.747639 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:19.774159 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3371db-7770-45c3-af1b-36f3f87e2831 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:19.825461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfb4d88-de74-4307-b8bb-03c0910ee6b3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:19.826057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfb4d88-de74-4307-b8bb-03c0910ee6b3 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:19.826327 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfb4d88-de74-4307-b8bb-03c0910ee6b3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:19.826588 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfb4d88-de74-4307-b8bb-03c0910ee6b3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:19.828461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9dfb4d88-de74-4307-b8bb-03c0910ee6b3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:19.844052 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dfb4d88-de74-4307-b8bb-03c0910ee6b3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:19.844052 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dfb4d88-de74-4307-b8bb-03c0910ee6b3 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:19.849198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1381/2755] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:19 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:37:19.929384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef3f889-166a-49bc-9b16-4d46c7bde3f0 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:19.931046 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-6ef3f889-166a-49bc-9b16-4d46c7bde3f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:20.008284 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef3f889-166a-49bc-9b16-4d46c7bde3f0 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:20.012973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9826ed77-3fa4-4497-b11a-eebb0fad5019 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:20.013628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9826ed77-3fa4-4497-b11a-eebb0fad5019 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:20.013905 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9826ed77-3fa4-4497-b11a-eebb0fad5019 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:20.014175 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9826ed77-3fa4-4497-b11a-eebb0fad5019 admin admin] PATH_INFO: `/v3/projects/863e6ce5f84842e98cdda3dfdb11f0c1` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:20.015312 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9826ed77-3fa4-4497-b11a-eebb0fad5019 admin admin] RBAC: Authorizing `identity:delete_project(project_id=863e6ce5f84842e98cdda3dfdb11f0c1)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:20.017281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9826ed77-3fa4-4497-b11a-eebb0fad5019 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:20.118416 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:20.122245 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-9826ed77-3fa4-4497-b11a-eebb0fad5019 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '863e6ce5f84842e98cdda3dfdb11f0c1', 'request_id': 'req-9826ed77-3fa4-4497-b11a-eebb0fad5019'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:20.127774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-9826ed77-3fa4-4497-b11a-eebb0fad5019 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '863e6ce5f84842e98cdda3dfdb11f0c1', 'request_id': 'req-9826ed77-3fa4-4497-b11a-eebb0fad5019'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:20.134687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-9826ed77-3fa4-4497-b11a-eebb0fad5019 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '863e6ce5f84842e98cdda3dfdb11f0c1', 'request_id': 'req-9826ed77-3fa4-4497-b11a-eebb0fad5019'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:20.188192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-9826ed77-3fa4-4497-b11a-eebb0fad5019 admin admin] The token cache is being invalidate because project 863e6ce5f84842e98cdda3dfdb11f0c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:37:20.188793 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-9826ed77-3fa4-4497-b11a-eebb0fad5019 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:20.190120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-9826ed77-3fa4-4497-b11a-eebb0fad5019 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:20.191133 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1375/2756] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:37:19 2022] DELETE /identity/v3/projects/863e6ce5f84842e98cdda3dfdb11f0c1 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:37:20.196368 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb461681-09da-4d64-b14f-990aab50fdf8 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:20.196646 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb461681-09da-4d64-b14f-990aab50fdf8 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:20.196908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb461681-09da-4d64-b14f-990aab50fdf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:20.217281 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-eb461681-09da-4d64-b14f-990aab50fdf8 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:20.218806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-eb461681-09da-4d64-b14f-990aab50fdf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:20.278843 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1382/2757] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:20.286878 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a1a1d8-2ee2-42fb-881f-2dc261f2f5c9 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:20.287628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a1a1d8-2ee2-42fb-881f-2dc261f2f5c9 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:20.287628 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a1a1d8-2ee2-42fb-881f-2dc261f2f5c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:20.307199 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-46a1a1d8-2ee2-42fb-881f-2dc261f2f5c9 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:20.308432 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-46a1a1d8-2ee2-42fb-881f-2dc261f2f5c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:20.316370 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1376/2758] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:37:20.321030 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb461681-09da-4d64-b14f-990aab50fdf8 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:20.330723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb461681-09da-4d64-b14f-990aab50fdf8 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:20.336295 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289d8378-8bb6-4093-8a28-04db8883819a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:20.336968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289d8378-8bb6-4093-8a28-04db8883819a admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:20.342055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289d8378-8bb6-4093-8a28-04db8883819a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:20.342055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289d8378-8bb6-4093-8a28-04db8883819a admin admin] PATH_INFO: `/v3/domains` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:20.342055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289d8378-8bb6-4093-8a28-04db8883819a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81804) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:37:20.342055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289d8378-8bb6-4093-8a28-04db8883819a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:20.342055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289d8378-8bb6-4093-8a28-04db8883819a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:20.354057 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1383/2759] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:37:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:37:20.358689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a1a1d8-2ee2-42fb-881f-2dc261f2f5c9 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:20.368690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a1a1d8-2ee2-42fb-881f-2dc261f2f5c9 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:20.373617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74517cd5-04ff-49fa-8c23-f466fe5e7a26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:20.374204 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74517cd5-04ff-49fa-8c23-f466fe5e7a26 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:20.374482 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74517cd5-04ff-49fa-8c23-f466fe5e7a26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:20.374751 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74517cd5-04ff-49fa-8c23-f466fe5e7a26 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:20.375418 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74517cd5-04ff-49fa-8c23-f466fe5e7a26 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:20.377325 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74517cd5-04ff-49fa-8c23-f466fe5e7a26 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:20.470130 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1377/2760] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:37:20 2022] POST /identity/v3/projects => generated 377 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:37:20.474300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289d8378-8bb6-4093-8a28-04db8883819a admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:20.484375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289d8378-8bb6-4093-8a28-04db8883819a admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:20.540393 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4a4372-839a-4677-93b6-ba1275c498a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:20.540986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4a4372-839a-4677-93b6-ba1275c498a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:20.541259 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4a4372-839a-4677-93b6-ba1275c498a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:20.541526 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4a4372-839a-4677-93b6-ba1275c498a3 admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:20.542167 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4a4372-839a-4677-93b6-ba1275c498a3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:20.544110 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4a4372-839a-4677-93b6-ba1275c498a3 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:20.596629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74517cd5-04ff-49fa-8c23-f466fe5e7a26 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:20.600346 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-74517cd5-04ff-49fa-8c23-f466fe5e7a26 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:20.664139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1384/2761] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:37:20 2022] POST /identity/v3/users => generated 289 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:20.668296 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4a4372-839a-4677-93b6-ba1275c498a3 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:20.673666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74517cd5-04ff-49fa-8c23-f466fe5e7a26 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:20.675086 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:20.679346 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4a4372-839a-4677-93b6-ba1275c498a3 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:20.679651 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415f73b2-b487-412a-afc8-dde54abe80ed service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '972b0f1dbfef4d85a6dfc4c5fd839bde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:20.680340 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415f73b2-b487-412a-afc8-dde54abe80ed service placement] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:20.680615 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415f73b2-b487-412a-afc8-dde54abe80ed service placement] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:20.680886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415f73b2-b487-412a-afc8-dde54abe80ed service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:20.682678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-415f73b2-b487-412a-afc8-dde54abe80ed service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:20.684054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742ff184-d17c-490f-8d2d-e47a5f29bdf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:20.684664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742ff184-d17c-490f-8d2d-e47a5f29bdf5 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:20.684935 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742ff184-d17c-490f-8d2d-e47a5f29bdf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:20.685198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742ff184-d17c-490f-8d2d-e47a5f29bdf5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:20.685825 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742ff184-d17c-490f-8d2d-e47a5f29bdf5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:20.687383 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742ff184-d17c-490f-8d2d-e47a5f29bdf5 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:20.695662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415f73b2-b487-412a-afc8-dde54abe80ed service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:20.697260 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415f73b2-b487-412a-afc8-dde54abe80ed service placement] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:20.697957 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1385/2762] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:37:20.702094 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742ff184-d17c-490f-8d2d-e47a5f29bdf5 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:20.707951 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1378/2763] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:20 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:37:20.711958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742ff184-d17c-490f-8d2d-e47a5f29bdf5 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:20.717303 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9554262-87be-4b45-976e-69227cd9d89d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:20.722960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9554262-87be-4b45-976e-69227cd9d89d admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:20.722960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9554262-87be-4b45-976e-69227cd9d89d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:20.722960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9554262-87be-4b45-976e-69227cd9d89d admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:20.722960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9554262-87be-4b45-976e-69227cd9d89d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:20.722960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9554262-87be-4b45-976e-69227cd9d89d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:20.736892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1386/2764] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:37:20.738075 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415f73b2-b487-412a-afc8-dde54abe80ed service placement] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:20.751609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415f73b2-b487-412a-afc8-dde54abe80ed service placement] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:20.753819 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b099ba65-ece2-4dfb-85e4-d47e286cd45d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:20.754488 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b099ba65-ece2-4dfb-85e4-d47e286cd45d admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:20.754761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b099ba65-ece2-4dfb-85e4-d47e286cd45d admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:20.755030 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b099ba65-ece2-4dfb-85e4-d47e286cd45d admin admin] PATH_INFO: `/v3/projects/91524e47334e4a25a28454c381811675/users/12744f8a64b441b0a6760f9a081ebe32/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:20.772766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b099ba65-ece2-4dfb-85e4-d47e286cd45d admin admin] RBAC: Authorizing `identity:create_grant(project_id=91524e47334e4a25a28454c381811675, user_id=12744f8a64b441b0a6760f9a081ebe32, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:20.775464 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b099ba65-ece2-4dfb-85e4-d47e286cd45d admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:20.841151 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1379/2765] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:37:20 2022] PUT /identity/v3/projects/91524e47334e4a25a28454c381811675/users/12744f8a64b441b0a6760f9a081ebe32/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:20.851942 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a81c3c9-2466-461b-8ce5-64bd35854d46 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:20.852194 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a81c3c9-2466-461b-8ce5-64bd35854d46 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:20.852465 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a81c3c9-2466-461b-8ce5-64bd35854d46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:20.879717 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-7a81c3c9-2466-461b-8ce5-64bd35854d46 None None] MFA Rules not processed for user `12744f8a64b441b0a6760f9a081ebe32`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:20.880947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-7a81c3c9-2466-461b-8ce5-64bd35854d46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:20.943529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1387/2766] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:20 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:20.948379 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b099ba65-ece2-4dfb-85e4-d47e286cd45d admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:21.009474 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b099ba65-ece2-4dfb-85e4-d47e286cd45d admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:21.014157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b51e0f0-8d86-4587-a257-8be8376c5d92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:21.014761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b51e0f0-8d86-4587-a257-8be8376c5d92 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:21.015038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b51e0f0-8d86-4587-a257-8be8376c5d92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:21.015306 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b51e0f0-8d86-4587-a257-8be8376c5d92 admin admin] PATH_INFO: `/v3/projects` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:21.015939 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b51e0f0-8d86-4587-a257-8be8376c5d92 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:21.017862 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b51e0f0-8d86-4587-a257-8be8376c5d92 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:21.034057 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:21.103484 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1380/2767] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:37:20 2022] POST /identity/v3/projects => generated 381 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:21.107454 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a81c3c9-2466-461b-8ce5-64bd35854d46 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:21.118236 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a81c3c9-2466-461b-8ce5-64bd35854d46 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:21.175689 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa0e90e-96f9-4b79-8a9e-aed5090f9abc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:21.176394 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa0e90e-96f9-4b79-8a9e-aed5090f9abc admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:21.176674 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa0e90e-96f9-4b79-8a9e-aed5090f9abc admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:21.176947 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa0e90e-96f9-4b79-8a9e-aed5090f9abc admin admin] PATH_INFO: `/v3/users` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:21.177610 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa0e90e-96f9-4b79-8a9e-aed5090f9abc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:21.179553 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa0e90e-96f9-4b79-8a9e-aed5090f9abc admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:21.268372 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1388/2768] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:37:21 2022] POST /identity/v3/users => generated 297 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:21.273038 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b51e0f0-8d86-4587-a257-8be8376c5d92 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:21.284321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b51e0f0-8d86-4587-a257-8be8376c5d92 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:21.290678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e565714b-ae7c-4f07-8307-7e5915078fd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:21.291349 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e565714b-ae7c-4f07-8307-7e5915078fd0 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:21.291677 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e565714b-ae7c-4f07-8307-7e5915078fd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:21.291994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e565714b-ae7c-4f07-8307-7e5915078fd0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:21.292784 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e565714b-ae7c-4f07-8307-7e5915078fd0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:21.294477 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e565714b-ae7c-4f07-8307-7e5915078fd0 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:21.305506 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1381/2769] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:21 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:37:21.309770 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa0e90e-96f9-4b79-8a9e-aed5090f9abc admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:21.319844 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa0e90e-96f9-4b79-8a9e-aed5090f9abc admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:21.324423 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f962a10d-00cd-43bd-a34c-797fa2f391e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:21.325072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f962a10d-00cd-43bd-a34c-797fa2f391e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:21.325324 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f962a10d-00cd-43bd-a34c-797fa2f391e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:21.325593 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f962a10d-00cd-43bd-a34c-797fa2f391e7 admin admin] PATH_INFO: `/v3/projects/9087275fc951472fa4a1b34b82a3c11f/users/136eaa0c87114b2a93c9c09eb75c4378/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:21.341119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f962a10d-00cd-43bd-a34c-797fa2f391e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9087275fc951472fa4a1b34b82a3c11f, user_id=136eaa0c87114b2a93c9c09eb75c4378, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:21.343824 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f962a10d-00cd-43bd-a34c-797fa2f391e7 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:21.405678 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1389/2770] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:37:21 2022] PUT /identity/v3/projects/9087275fc951472fa4a1b34b82a3c11f/users/136eaa0c87114b2a93c9c09eb75c4378/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:21.409791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e565714b-ae7c-4f07-8307-7e5915078fd0 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:21.420314 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e565714b-ae7c-4f07-8307-7e5915078fd0 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:21.476364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62dd9763-f36f-426c-b29b-72197bdca61f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:21.476964 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62dd9763-f36f-426c-b29b-72197bdca61f admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:21.477243 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62dd9763-f36f-426c-b29b-72197bdca61f admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:21.477510 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62dd9763-f36f-426c-b29b-72197bdca61f admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:21.478136 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62dd9763-f36f-426c-b29b-72197bdca61f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:21.479722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62dd9763-f36f-426c-b29b-72197bdca61f admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:21.489481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1382/2771] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:21 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:37:21.493648 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f962a10d-00cd-43bd-a34c-797fa2f391e7 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:21.503949 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f962a10d-00cd-43bd-a34c-797fa2f391e7 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:21.509154 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9e1232-93ed-431e-b6e4-12fc5b9f9492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:21.509798 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9e1232-93ed-431e-b6e4-12fc5b9f9492 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:21.510075 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9e1232-93ed-431e-b6e4-12fc5b9f9492 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:21.510346 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9e1232-93ed-431e-b6e4-12fc5b9f9492 admin admin] PATH_INFO: `/v3/projects/9087275fc951472fa4a1b34b82a3c11f/users/136eaa0c87114b2a93c9c09eb75c4378/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:21.512297 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9e1232-93ed-431e-b6e4-12fc5b9f9492 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9087275fc951472fa4a1b34b82a3c11f, user_id=136eaa0c87114b2a93c9c09eb75c4378, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:21.514895 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9e1232-93ed-431e-b6e4-12fc5b9f9492 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:21.525592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1390/2772] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:37:21 2022] PUT /identity/v3/projects/9087275fc951472fa4a1b34b82a3c11f/users/136eaa0c87114b2a93c9c09eb75c4378/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:21.590475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a7f708-b615-40b5-b2de-a2b2b8e98a3c None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:21.590756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a7f708-b615-40b5-b2de-a2b2b8e98a3c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:21.591028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a7f708-b615-40b5-b2de-a2b2b8e98a3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:21.619182 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-37a7f708-b615-40b5-b2de-a2b2b8e98a3c None None] MFA Rules not processed for user `136eaa0c87114b2a93c9c09eb75c4378`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:21.620342 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-37a7f708-b615-40b5-b2de-a2b2b8e98a3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:21.690226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1383/2773] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:21 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:21.702502 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9e1232-93ed-431e-b6e4-12fc5b9f9492 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:21.704278 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-fc9e1232-93ed-431e-b6e4-12fc5b9f9492 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:21.735742 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:21.766476 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9e1232-93ed-431e-b6e4-12fc5b9f9492 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:21.771901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78556f3-1a81-4ea9-845c-add9379aa783 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:21.772056 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78556f3-1a81-4ea9-845c-add9379aa783 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:21.773069 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78556f3-1a81-4ea9-845c-add9379aa783 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:21.773069 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78556f3-1a81-4ea9-845c-add9379aa783 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:21.782569 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78556f3-1a81-4ea9-845c-add9379aa783 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:21.783993 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78556f3-1a81-4ea9-845c-add9379aa783 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:21.793557 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1391/2774] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:21 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:37:21.963453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a7f708-b615-40b5-b2de-a2b2b8e98a3c None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:21.973639 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a7f708-b615-40b5-b2de-a2b2b8e98a3c None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:21.977270 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da582eb0-0e1b-433a-90a5-7b177bebc6a5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:21.977901 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da582eb0-0e1b-433a-90a5-7b177bebc6a5 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:21.978181 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da582eb0-0e1b-433a-90a5-7b177bebc6a5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:21.978449 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da582eb0-0e1b-433a-90a5-7b177bebc6a5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:22.035881 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da582eb0-0e1b-433a-90a5-7b177bebc6a5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:22.037358 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da582eb0-0e1b-433a-90a5-7b177bebc6a5 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:22.046636 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1384/2775] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:21 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:37:31.799744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=15 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:32.042360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:INSERT=1,keystone:SELECT=98 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:33.803322 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78556f3-1a81-4ea9-845c-add9379aa783 service cinder] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:33.887840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78556f3-1a81-4ea9-845c-add9379aa783 service cinder] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:33.892666 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f15eba4-59cb-4040-932c-55d6e1a96a11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:33.893307 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f15eba4-59cb-4040-932c-55d6e1a96a11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:33.893591 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f15eba4-59cb-4040-932c-55d6e1a96a11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:33.893857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f15eba4-59cb-4040-932c-55d6e1a96a11 admin admin] PATH_INFO: `/v3/users/12744f8a64b441b0a6760f9a081ebe32` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:33.895555 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f15eba4-59cb-4040-932c-55d6e1a96a11 admin admin] RBAC: Authorizing `identity:delete_user(user_id=12744f8a64b441b0a6760f9a081ebe32)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:33.897506 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f15eba4-59cb-4040-932c-55d6e1a96a11 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:34.087276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8f15eba4-59cb-4040-932c-55d6e1a96a11 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '12744f8a64b441b0a6760f9a081ebe32'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:34.094307 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8f15eba4-59cb-4040-932c-55d6e1a96a11 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '12744f8a64b441b0a6760f9a081ebe32'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:34.107224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8f15eba4-59cb-4040-932c-55d6e1a96a11 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '12744f8a64b441b0a6760f9a081ebe32'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:34.176657 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8f15eba4-59cb-4040-932c-55d6e1a96a11 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '12744f8a64b441b0a6760f9a081ebe32'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:34.177279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8f15eba4-59cb-4040-932c-55d6e1a96a11 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '12744f8a64b441b0a6760f9a081ebe32'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:34.183660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1392/2776] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:37:33 2022] DELETE /identity/v3/users/12744f8a64b441b0a6760f9a081ebe32 => generated 0 bytes in 381 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:34.188228 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea06af8f-e203-4961-9c62-f1c42e41292a None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:34.188746 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea06af8f-e203-4961-9c62-f1c42e41292a None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:34.188746 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea06af8f-e203-4961-9c62-f1c42e41292a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:34.208876 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-ea06af8f-e203-4961-9c62-f1c42e41292a None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:34.210353 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ea06af8f-e203-4961-9c62-f1c42e41292a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:34.268448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1385/2777] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:34.278954 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f15eba4-59cb-4040-932c-55d6e1a96a11 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:34.280393 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-8f15eba4-59cb-4040-932c-55d6e1a96a11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:34.336772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f15eba4-59cb-4040-932c-55d6e1a96a11 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:34.340460 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7250bf6-8084-4fda-87c3-088ed9ce9638 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:34.341095 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7250bf6-8084-4fda-87c3-088ed9ce9638 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:34.341378 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7250bf6-8084-4fda-87c3-088ed9ce9638 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:34.341633 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7250bf6-8084-4fda-87c3-088ed9ce9638 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:34.351254 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7250bf6-8084-4fda-87c3-088ed9ce9638 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:34.352603 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7250bf6-8084-4fda-87c3-088ed9ce9638 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:34.362169 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1393/2778] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:37:34.841886 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea06af8f-e203-4961-9c62-f1c42e41292a None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:34.843411 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ea06af8f-e203-4961-9c62-f1c42e41292a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:34.855960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea06af8f-e203-4961-9c62-f1c42e41292a None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:34.861022 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31df7d35-5b3a-4e16-b5a0-7901774a1715 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:34.861717 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df7d35-5b3a-4e16-b5a0-7901774a1715 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:34.861989 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df7d35-5b3a-4e16-b5a0-7901774a1715 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:34.862252 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df7d35-5b3a-4e16-b5a0-7901774a1715 admin admin] PATH_INFO: `/v3/projects/91524e47334e4a25a28454c381811675` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:34.864508 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31df7d35-5b3a-4e16-b5a0-7901774a1715 admin admin] RBAC: Authorizing `identity:delete_project(project_id=91524e47334e4a25a28454c381811675)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:34.871321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31df7d35-5b3a-4e16-b5a0-7901774a1715 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:34.981405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-31df7d35-5b3a-4e16-b5a0-7901774a1715 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91524e47334e4a25a28454c381811675', 'request_id': 'req-31df7d35-5b3a-4e16-b5a0-7901774a1715'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:34.988063 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-31df7d35-5b3a-4e16-b5a0-7901774a1715 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91524e47334e4a25a28454c381811675', 'request_id': 'req-31df7d35-5b3a-4e16-b5a0-7901774a1715'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:34.995513 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-31df7d35-5b3a-4e16-b5a0-7901774a1715 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91524e47334e4a25a28454c381811675', 'request_id': 'req-31df7d35-5b3a-4e16-b5a0-7901774a1715'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:35.079054 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-31df7d35-5b3a-4e16-b5a0-7901774a1715 admin admin] The token cache is being invalidate because project 91524e47334e4a25a28454c381811675 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:37:35.079338 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-31df7d35-5b3a-4e16-b5a0-7901774a1715 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:35.079964 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-31df7d35-5b3a-4e16-b5a0-7901774a1715 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:35.081674 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1386/2779] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:37:34 2022] DELETE /identity/v3/projects/91524e47334e4a25a28454c381811675 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:35.089894 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7250bf6-8084-4fda-87c3-088ed9ce9638 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:35.096503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7250bf6-8084-4fda-87c3-088ed9ce9638 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:35.120119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:35.160886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f25614-209d-475f-97ad-d13220662409 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:35.161624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f25614-209d-475f-97ad-d13220662409 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:35.164503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f25614-209d-475f-97ad-d13220662409 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:35.164503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f25614-209d-475f-97ad-d13220662409 admin admin] PATH_INFO: `/v3/users/136eaa0c87114b2a93c9c09eb75c4378` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:35.164503 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f25614-209d-475f-97ad-d13220662409 admin admin] RBAC: Authorizing `identity:delete_user(user_id=136eaa0c87114b2a93c9c09eb75c4378)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:35.165411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f25614-209d-475f-97ad-d13220662409 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:35.338975 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-33f25614-209d-475f-97ad-d13220662409 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '136eaa0c87114b2a93c9c09eb75c4378'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:35.355019 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-33f25614-209d-475f-97ad-d13220662409 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '136eaa0c87114b2a93c9c09eb75c4378'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:35.355191 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31df7d35-5b3a-4e16-b5a0-7901774a1715 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:35.356678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-31df7d35-5b3a-4e16-b5a0-7901774a1715 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:35.369205 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-33f25614-209d-475f-97ad-d13220662409 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '136eaa0c87114b2a93c9c09eb75c4378'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:35.430354 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31df7d35-5b3a-4e16-b5a0-7901774a1715 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:35.435860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12435291-c9cc-4a34-bd89-f02c1a577e3b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:35.436486 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12435291-c9cc-4a34-bd89-f02c1a577e3b service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:35.436760 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12435291-c9cc-4a34-bd89-f02c1a577e3b service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:35.437276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12435291-c9cc-4a34-bd89-f02c1a577e3b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:35.439104 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-12435291-c9cc-4a34-bd89-f02c1a577e3b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:35.441594 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-33f25614-209d-475f-97ad-d13220662409 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '136eaa0c87114b2a93c9c09eb75c4378'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:35.442460 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-33f25614-209d-475f-97ad-d13220662409 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '136eaa0c87114b2a93c9c09eb75c4378'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:35.454282 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1394/2780] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:37:35 2022] DELETE /identity/v3/users/136eaa0c87114b2a93c9c09eb75c4378 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:35.454478 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12435291-c9cc-4a34-bd89-f02c1a577e3b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:35.454478 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12435291-c9cc-4a34-bd89-f02c1a577e3b service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:35.468880 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1387/2781] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:37:35.843223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f25614-209d-475f-97ad-d13220662409 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:35.844709 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-33f25614-209d-475f-97ad-d13220662409 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:35.920662 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f25614-209d-475f-97ad-d13220662409 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:35.926820 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a006fd-fe25-476d-9918-189fc956269b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:35.928050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a006fd-fe25-476d-9918-189fc956269b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:35.928519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a006fd-fe25-476d-9918-189fc956269b admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:35.929000 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a006fd-fe25-476d-9918-189fc956269b admin admin] PATH_INFO: `/v3/projects/9087275fc951472fa4a1b34b82a3c11f` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:35.930285 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a006fd-fe25-476d-9918-189fc956269b admin admin] RBAC: Authorizing `identity:delete_project(project_id=9087275fc951472fa4a1b34b82a3c11f)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:35.932240 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a006fd-fe25-476d-9918-189fc956269b admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:36.084871 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:36.088716 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-60a006fd-fe25-476d-9918-189fc956269b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9087275fc951472fa4a1b34b82a3c11f', 'request_id': 'req-60a006fd-fe25-476d-9918-189fc956269b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:36.147605 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-60a006fd-fe25-476d-9918-189fc956269b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9087275fc951472fa4a1b34b82a3c11f', 'request_id': 'req-60a006fd-fe25-476d-9918-189fc956269b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:36.153826 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-60a006fd-fe25-476d-9918-189fc956269b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9087275fc951472fa4a1b34b82a3c11f', 'request_id': 'req-60a006fd-fe25-476d-9918-189fc956269b'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:36.165479 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-60a006fd-fe25-476d-9918-189fc956269b admin admin] The token cache is being invalidate because project 9087275fc951472fa4a1b34b82a3c11f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:37:36.165763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-60a006fd-fe25-476d-9918-189fc956269b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:36.166411 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-60a006fd-fe25-476d-9918-189fc956269b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:36.168043 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1395/2782] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:37:35 2022] DELETE /identity/v3/projects/9087275fc951472fa4a1b34b82a3c11f => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:36.172960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075710d1-bbaf-47ad-8c52-7a88d9958ba8 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:36.173345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075710d1-bbaf-47ad-8c52-7a88d9958ba8 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:36.173625 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075710d1-bbaf-47ad-8c52-7a88d9958ba8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:36.194547 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-075710d1-bbaf-47ad-8c52-7a88d9958ba8 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:36.195765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-075710d1-bbaf-47ad-8c52-7a88d9958ba8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:36.200202 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:36.260343 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1388/2783] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:36.268017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd9050c-4cee-4a8b-9959-3296447159dc None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:36.268333 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd9050c-4cee-4a8b-9959-3296447159dc None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:36.268626 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd9050c-4cee-4a8b-9959-3296447159dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:36.289062 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-bbd9050c-4cee-4a8b-9959-3296447159dc None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:36.290407 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-bbd9050c-4cee-4a8b-9959-3296447159dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:36.298850 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1396/2784] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:36.303614 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075710d1-bbaf-47ad-8c52-7a88d9958ba8 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:36.313679 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075710d1-bbaf-47ad-8c52-7a88d9958ba8 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:36.319261 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e8f02b-c974-4687-90cd-5c7045ad0643 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:36.319858 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e8f02b-c974-4687-90cd-5c7045ad0643 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:36.320160 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e8f02b-c974-4687-90cd-5c7045ad0643 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:36.320435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e8f02b-c974-4687-90cd-5c7045ad0643 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:36.321075 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e8f02b-c974-4687-90cd-5c7045ad0643 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:37:36.321376 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e8f02b-c974-4687-90cd-5c7045ad0643 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:36.323112 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e8f02b-c974-4687-90cd-5c7045ad0643 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:36.335722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1389/2785] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:37:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:37:36.339589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd9050c-4cee-4a8b-9959-3296447159dc None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:36.348895 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd9050c-4cee-4a8b-9959-3296447159dc None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:36.353599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b65991-bbc8-43f0-bb14-53b0dbaae561 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:36.354192 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b65991-bbc8-43f0-bb14-53b0dbaae561 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:36.354461 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b65991-bbc8-43f0-bb14-53b0dbaae561 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:36.354723 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b65991-bbc8-43f0-bb14-53b0dbaae561 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:36.355333 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b65991-bbc8-43f0-bb14-53b0dbaae561 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:36.357385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b65991-bbc8-43f0-bb14-53b0dbaae561 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:36.469499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1397/2786] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:37:36 2022] POST /identity/v3/projects => generated 365 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:36.474624 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e8f02b-c974-4687-90cd-5c7045ad0643 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:36.485148 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e8f02b-c974-4687-90cd-5c7045ad0643 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:36.540475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e6e3a5-e647-45ae-b4d9-4c1ee7c1f91e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:36.541081 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e6e3a5-e647-45ae-b4d9-4c1ee7c1f91e admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:36.541379 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e6e3a5-e647-45ae-b4d9-4c1ee7c1f91e admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:36.541631 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e6e3a5-e647-45ae-b4d9-4c1ee7c1f91e admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:36.542245 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e6e3a5-e647-45ae-b4d9-4c1ee7c1f91e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:36.544167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e6e3a5-e647-45ae-b4d9-4c1ee7c1f91e admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:36.633231 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1390/2787] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:37:36 2022] POST /identity/v3/users => generated 283 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:36.637385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b65991-bbc8-43f0-bb14-53b0dbaae561 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:36.647908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b65991-bbc8-43f0-bb14-53b0dbaae561 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:36.652585 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79155d4c-6746-4d8c-818d-2eb0f8d3dcb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:36.653179 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79155d4c-6746-4d8c-818d-2eb0f8d3dcb0 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:36.653452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79155d4c-6746-4d8c-818d-2eb0f8d3dcb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:36.653716 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79155d4c-6746-4d8c-818d-2eb0f8d3dcb0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:36.654354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79155d4c-6746-4d8c-818d-2eb0f8d3dcb0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:36.655933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79155d4c-6746-4d8c-818d-2eb0f8d3dcb0 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:36.666249 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1398/2788] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:36 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:37:36.670193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e6e3a5-e647-45ae-b4d9-4c1ee7c1f91e admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:36.681384 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e6e3a5-e647-45ae-b4d9-4c1ee7c1f91e admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:36.686065 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08da148b-6218-4ebc-b05f-3befbd9189fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:36.686659 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08da148b-6218-4ebc-b05f-3befbd9189fe admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:36.686986 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08da148b-6218-4ebc-b05f-3befbd9189fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:36.687245 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08da148b-6218-4ebc-b05f-3befbd9189fe admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:36.687870 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08da148b-6218-4ebc-b05f-3befbd9189fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:36.689514 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08da148b-6218-4ebc-b05f-3befbd9189fe admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:36.700730 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1391/2789] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:37:36.704793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79155d4c-6746-4d8c-818d-2eb0f8d3dcb0 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:36.715294 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79155d4c-6746-4d8c-818d-2eb0f8d3dcb0 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:36.720147 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9462f72-fbf8-471a-9bb9-0502c79b28f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:36.720740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9462f72-fbf8-471a-9bb9-0502c79b28f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:36.721013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9462f72-fbf8-471a-9bb9-0502c79b28f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:36.721300 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9462f72-fbf8-471a-9bb9-0502c79b28f1 admin admin] PATH_INFO: `/v3/projects/39616f87f21e4f10bb72b02929a4a492/users/2440670b0e0c4f0bbe79af7eb93208ae/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:36.737402 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9462f72-fbf8-471a-9bb9-0502c79b28f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=39616f87f21e4f10bb72b02929a4a492, user_id=2440670b0e0c4f0bbe79af7eb93208ae, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:36.742771 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9462f72-fbf8-471a-9bb9-0502c79b28f1 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:36.787811 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1399/2790] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:37:36 2022] PUT /identity/v3/projects/39616f87f21e4f10bb72b02929a4a492/users/2440670b0e0c4f0bbe79af7eb93208ae/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:36.799276 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ab43df-42ae-4d39-a980-f95389e15da2 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:36.799560 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ab43df-42ae-4d39-a980-f95389e15da2 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:36.799826 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ab43df-42ae-4d39-a980-f95389e15da2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:36.828163 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-47ab43df-42ae-4d39-a980-f95389e15da2 None None] MFA Rules not processed for user `2440670b0e0c4f0bbe79af7eb93208ae`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:36.829366 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-47ab43df-42ae-4d39-a980-f95389e15da2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:36.892850 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1392/2791] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:36 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:36.898150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9462f72-fbf8-471a-9bb9-0502c79b28f1 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:36.961836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9462f72-fbf8-471a-9bb9-0502c79b28f1 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:36.966592 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3986ea-3588-4c25-b569-80bc6612619f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:36.967188 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3986ea-3588-4c25-b569-80bc6612619f admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:36.967467 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3986ea-3588-4c25-b569-80bc6612619f admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:36.967741 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3986ea-3588-4c25-b569-80bc6612619f admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:36.968409 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3986ea-3588-4c25-b569-80bc6612619f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:36.971747 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3986ea-3588-4c25-b569-80bc6612619f admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:37.092421 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1400/2792] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:37:36 2022] POST /identity/v3/projects => generated 365 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:37.096709 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ab43df-42ae-4d39-a980-f95389e15da2 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:37.104611 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:37.108649 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ab43df-42ae-4d39-a980-f95389e15da2 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:37.172363 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c8069a-ae46-4751-9eee-37a24ae4cce2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:37.177092 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c8069a-ae46-4751-9eee-37a24ae4cce2 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:37.177442 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c8069a-ae46-4751-9eee-37a24ae4cce2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:37.177810 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c8069a-ae46-4751-9eee-37a24ae4cce2 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:37.178518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c8069a-ae46-4751-9eee-37a24ae4cce2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:37.182377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c8069a-ae46-4751-9eee-37a24ae4cce2 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:37.277985 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1393/2793] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:37:37 2022] POST /identity/v3/users => generated 289 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:37:37.279170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3986ea-3588-4c25-b569-80bc6612619f admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:37.290136 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3986ea-3588-4c25-b569-80bc6612619f admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:37.294779 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d4cb3d-bce5-4a63-ab81-3f72469d5d04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:37.295386 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d4cb3d-bce5-4a63-ab81-3f72469d5d04 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:37.295660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d4cb3d-bce5-4a63-ab81-3f72469d5d04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:37.295923 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d4cb3d-bce5-4a63-ab81-3f72469d5d04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:37.296614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d4cb3d-bce5-4a63-ab81-3f72469d5d04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:37.298189 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d4cb3d-bce5-4a63-ab81-3f72469d5d04 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:37.309772 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1401/2794] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:37 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:37:37.314110 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c8069a-ae46-4751-9eee-37a24ae4cce2 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:37.325313 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c8069a-ae46-4751-9eee-37a24ae4cce2 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:37.330321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5e70e0-0891-4f34-a263-41ff83cfe6ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:37.330990 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5e70e0-0891-4f34-a263-41ff83cfe6ee admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:37.331304 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5e70e0-0891-4f34-a263-41ff83cfe6ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:37.331746 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5e70e0-0891-4f34-a263-41ff83cfe6ee admin admin] PATH_INFO: `/v3/projects/682b9d24e2ac4815bff836ca51ef3780/users/7ff0aa1e1b3249e490213fae65b04f64/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:37.350338 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5e70e0-0891-4f34-a263-41ff83cfe6ee admin admin] RBAC: Authorizing `identity:create_grant(project_id=682b9d24e2ac4815bff836ca51ef3780, user_id=7ff0aa1e1b3249e490213fae65b04f64, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:37.353207 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5e70e0-0891-4f34-a263-41ff83cfe6ee admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:37.422620 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1394/2795] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:37:37 2022] PUT /identity/v3/projects/682b9d24e2ac4815bff836ca51ef3780/users/7ff0aa1e1b3249e490213fae65b04f64/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:37.426930 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d4cb3d-bce5-4a63-ab81-3f72469d5d04 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:37.437079 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d4cb3d-bce5-4a63-ab81-3f72469d5d04 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:37.487660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:37.506854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5097a4db-5d02-4804-bf72-1cdcf0cfa5ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:37.507353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5097a4db-5d02-4804-bf72-1cdcf0cfa5ac admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:37.507627 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5097a4db-5d02-4804-bf72-1cdcf0cfa5ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:37.507937 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5097a4db-5d02-4804-bf72-1cdcf0cfa5ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:37.508565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5097a4db-5d02-4804-bf72-1cdcf0cfa5ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:37.510170 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5097a4db-5d02-4804-bf72-1cdcf0cfa5ac admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:37.522029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1402/2796] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:37 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:37:37.526099 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5e70e0-0891-4f34-a263-41ff83cfe6ee admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:37.538318 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5e70e0-0891-4f34-a263-41ff83cfe6ee admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:37.542985 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf237d5a-64db-4804-9ba7-00a30dab1c38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:37.543692 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf237d5a-64db-4804-9ba7-00a30dab1c38 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:37.543974 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf237d5a-64db-4804-9ba7-00a30dab1c38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:37.544281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf237d5a-64db-4804-9ba7-00a30dab1c38 admin admin] PATH_INFO: `/v3/projects/682b9d24e2ac4815bff836ca51ef3780/users/7ff0aa1e1b3249e490213fae65b04f64/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:37.546376 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf237d5a-64db-4804-9ba7-00a30dab1c38 admin admin] RBAC: Authorizing `identity:create_grant(project_id=682b9d24e2ac4815bff836ca51ef3780, user_id=7ff0aa1e1b3249e490213fae65b04f64, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:37.549102 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf237d5a-64db-4804-9ba7-00a30dab1c38 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:37.563291 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1395/2797] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:37:37 2022] PUT /identity/v3/projects/682b9d24e2ac4815bff836ca51ef3780/users/7ff0aa1e1b3249e490213fae65b04f64/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:37:37.574924 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f856e6-f750-4fb8-8577-0667f2bac92d None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:37.575235 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f856e6-f750-4fb8-8577-0667f2bac92d None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:37.575612 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f856e6-f750-4fb8-8577-0667f2bac92d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:37.606403 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-f2f856e6-f750-4fb8-8577-0667f2bac92d None None] MFA Rules not processed for user `7ff0aa1e1b3249e490213fae65b04f64`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:37.607664 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-f2f856e6-f750-4fb8-8577-0667f2bac92d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:37.680131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1403/2798] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:37 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:37.692681 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf237d5a-64db-4804-9ba7-00a30dab1c38 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:37.694426 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-bf237d5a-64db-4804-9ba7-00a30dab1c38 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:37.761714 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf237d5a-64db-4804-9ba7-00a30dab1c38 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:37.765568 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8976a90-2200-40af-8f22-4e6bf75822f3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:37.766177 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8976a90-2200-40af-8f22-4e6bf75822f3 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:37.766478 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8976a90-2200-40af-8f22-4e6bf75822f3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:37.766749 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8976a90-2200-40af-8f22-4e6bf75822f3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:37.777086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8976a90-2200-40af-8f22-4e6bf75822f3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:37.778519 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8976a90-2200-40af-8f22-4e6bf75822f3 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:37.788909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1396/2799] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:37 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:37:38.710494 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f856e6-f750-4fb8-8577-0667f2bac92d None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:38.778484 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f856e6-f750-4fb8-8577-0667f2bac92d None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:38.783687 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523c3a46-dc24-46e6-9765-e756a07ae487 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:38.786273 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523c3a46-dc24-46e6-9765-e756a07ae487 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:38.786273 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523c3a46-dc24-46e6-9765-e756a07ae487 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:38.786273 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523c3a46-dc24-46e6-9765-e756a07ae487 admin admin] PATH_INFO: `/v3/users/2440670b0e0c4f0bbe79af7eb93208ae` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:38.786273 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523c3a46-dc24-46e6-9765-e756a07ae487 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2440670b0e0c4f0bbe79af7eb93208ae)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:38.788148 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523c3a46-dc24-46e6-9765-e756a07ae487 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:38.973862 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-523c3a46-dc24-46e6-9765-e756a07ae487 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2440670b0e0c4f0bbe79af7eb93208ae'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:38.981538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-523c3a46-dc24-46e6-9765-e756a07ae487 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2440670b0e0c4f0bbe79af7eb93208ae'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:38.988450 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:38.993958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-523c3a46-dc24-46e6-9765-e756a07ae487 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2440670b0e0c4f0bbe79af7eb93208ae'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:39.056081 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-523c3a46-dc24-46e6-9765-e756a07ae487 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2440670b0e0c4f0bbe79af7eb93208ae'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:39.056782 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-523c3a46-dc24-46e6-9765-e756a07ae487 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2440670b0e0c4f0bbe79af7eb93208ae'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:39.063458 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1404/2800] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:37:38 2022] DELETE /identity/v3/users/2440670b0e0c4f0bbe79af7eb93208ae => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:39.068904 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f476c77c-40da-4257-92a1-16c1f5eeb2f1 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:39.069784 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f476c77c-40da-4257-92a1-16c1f5eeb2f1 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:39.069969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f476c77c-40da-4257-92a1-16c1f5eeb2f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:39.090983 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f476c77c-40da-4257-92a1-16c1f5eeb2f1 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:39.092424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f476c77c-40da-4257-92a1-16c1f5eeb2f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:39.145727 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:39.154143 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1397/2801] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:37:39.164131 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523c3a46-dc24-46e6-9765-e756a07ae487 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:39.165599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-523c3a46-dc24-46e6-9765-e756a07ae487 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:39.230357 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523c3a46-dc24-46e6-9765-e756a07ae487 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:39.234181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a54a2c-6eb1-488b-905f-ff17384e4553 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:39.234785 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a54a2c-6eb1-488b-905f-ff17384e4553 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:39.235054 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a54a2c-6eb1-488b-905f-ff17384e4553 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:39.235321 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a54a2c-6eb1-488b-905f-ff17384e4553 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:39.245659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a54a2c-6eb1-488b-905f-ff17384e4553 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:39.247120 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a54a2c-6eb1-488b-905f-ff17384e4553 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:39.258272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1405/2802] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:37:39.556603 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f476c77c-40da-4257-92a1-16c1f5eeb2f1 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:39.558016 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f476c77c-40da-4257-92a1-16c1f5eeb2f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:39.636952 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f476c77c-40da-4257-92a1-16c1f5eeb2f1 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:39.642596 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de58d243-c352-4222-afc8-cdc25a8e1063 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:39.643217 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de58d243-c352-4222-afc8-cdc25a8e1063 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:39.643491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de58d243-c352-4222-afc8-cdc25a8e1063 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:39.643760 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de58d243-c352-4222-afc8-cdc25a8e1063 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:39.645600 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-de58d243-c352-4222-afc8-cdc25a8e1063 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:39.657994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de58d243-c352-4222-afc8-cdc25a8e1063 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:39.659873 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de58d243-c352-4222-afc8-cdc25a8e1063 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:39.675431 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a54a2c-6eb1-488b-905f-ff17384e4553 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:39.675590 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1398/2803] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:39 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:37:39.675758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-25a54a2c-6eb1-488b-905f-ff17384e4553 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:39.687928 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a54a2c-6eb1-488b-905f-ff17384e4553 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:39.708023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f073829-b870-44a3-90f8-2b6e98045fa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:39.709072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f073829-b870-44a3-90f8-2b6e98045fa8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:39.710347 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f073829-b870-44a3-90f8-2b6e98045fa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:39.710561 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f073829-b870-44a3-90f8-2b6e98045fa8 admin admin] PATH_INFO: `/v3/projects/39616f87f21e4f10bb72b02929a4a492` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:39.712519 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f073829-b870-44a3-90f8-2b6e98045fa8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=39616f87f21e4f10bb72b02929a4a492)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:39.715780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f073829-b870-44a3-90f8-2b6e98045fa8 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:39.837262 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8f073829-b870-44a3-90f8-2b6e98045fa8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39616f87f21e4f10bb72b02929a4a492', 'request_id': 'req-8f073829-b870-44a3-90f8-2b6e98045fa8'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:39.891331 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8f073829-b870-44a3-90f8-2b6e98045fa8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39616f87f21e4f10bb72b02929a4a492', 'request_id': 'req-8f073829-b870-44a3-90f8-2b6e98045fa8'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:39.900050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8f073829-b870-44a3-90f8-2b6e98045fa8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39616f87f21e4f10bb72b02929a4a492', 'request_id': 'req-8f073829-b870-44a3-90f8-2b6e98045fa8'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:39.913254 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8f073829-b870-44a3-90f8-2b6e98045fa8 admin admin] The token cache is being invalidate because project 39616f87f21e4f10bb72b02929a4a492 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:37:39.913538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8f073829-b870-44a3-90f8-2b6e98045fa8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:39.914248 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-8f073829-b870-44a3-90f8-2b6e98045fa8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:39.916259 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1406/2804] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:37:39 2022] DELETE /identity/v3/projects/39616f87f21e4f10bb72b02929a4a492 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:39.920282 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de58d243-c352-4222-afc8-cdc25a8e1063 service nova] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:39.931731 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de58d243-c352-4222-afc8-cdc25a8e1063 service nova] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:40.000671 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f11f8d0-0bb3-4cbe-a2dd-ae79684694f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:40.001430 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f11f8d0-0bb3-4cbe-a2dd-ae79684694f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:40.001809 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f11f8d0-0bb3-4cbe-a2dd-ae79684694f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:40.002180 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f11f8d0-0bb3-4cbe-a2dd-ae79684694f1 admin admin] PATH_INFO: `/v3/users/7ff0aa1e1b3249e490213fae65b04f64` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:40.003428 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f11f8d0-0bb3-4cbe-a2dd-ae79684694f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7ff0aa1e1b3249e490213fae65b04f64)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:40.007438 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f11f8d0-0bb3-4cbe-a2dd-ae79684694f1 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:40.139571 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-8f11f8d0-0bb3-4cbe-a2dd-ae79684694f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7ff0aa1e1b3249e490213fae65b04f64'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:40.148954 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-8f11f8d0-0bb3-4cbe-a2dd-ae79684694f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7ff0aa1e1b3249e490213fae65b04f64'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:40.149850 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-8f11f8d0-0bb3-4cbe-a2dd-ae79684694f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7ff0aa1e1b3249e490213fae65b04f64'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:40.169316 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-8f11f8d0-0bb3-4cbe-a2dd-ae79684694f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7ff0aa1e1b3249e490213fae65b04f64'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:40.222134 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-8f11f8d0-0bb3-4cbe-a2dd-ae79684694f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7ff0aa1e1b3249e490213fae65b04f64'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:40.244223 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1399/2805] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:37:39 2022] DELETE /identity/v3/users/7ff0aa1e1b3249e490213fae65b04f64 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:37:40.658163 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f073829-b870-44a3-90f8-2b6e98045fa8 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:40.659668 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-8f073829-b870-44a3-90f8-2b6e98045fa8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:40.673611 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f073829-b870-44a3-90f8-2b6e98045fa8 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:40.734392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97673aa5-39b2-4ec5-9b40-2ac374e5b927 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:40.735005 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97673aa5-39b2-4ec5-9b40-2ac374e5b927 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:40.735005 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97673aa5-39b2-4ec5-9b40-2ac374e5b927 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:40.735005 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97673aa5-39b2-4ec5-9b40-2ac374e5b927 admin admin] PATH_INFO: `/v3/projects/682b9d24e2ac4815bff836ca51ef3780` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:40.736084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97673aa5-39b2-4ec5-9b40-2ac374e5b927 admin admin] RBAC: Authorizing `identity:delete_project(project_id=682b9d24e2ac4815bff836ca51ef3780)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:40.738004 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97673aa5-39b2-4ec5-9b40-2ac374e5b927 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:40.748286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:40.873301 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-97673aa5-39b2-4ec5-9b40-2ac374e5b927 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '682b9d24e2ac4815bff836ca51ef3780', 'request_id': 'req-97673aa5-39b2-4ec5-9b40-2ac374e5b927'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:40.947805 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-97673aa5-39b2-4ec5-9b40-2ac374e5b927 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '682b9d24e2ac4815bff836ca51ef3780', 'request_id': 'req-97673aa5-39b2-4ec5-9b40-2ac374e5b927'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:40.954078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-97673aa5-39b2-4ec5-9b40-2ac374e5b927 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '682b9d24e2ac4815bff836ca51ef3780', 'request_id': 'req-97673aa5-39b2-4ec5-9b40-2ac374e5b927'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:40.966215 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-97673aa5-39b2-4ec5-9b40-2ac374e5b927 admin admin] The token cache is being invalidate because project 682b9d24e2ac4815bff836ca51ef3780 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:37:40.966498 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-97673aa5-39b2-4ec5-9b40-2ac374e5b927 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:40.967136 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-97673aa5-39b2-4ec5-9b40-2ac374e5b927 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:40.968712 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1407/2806] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:37:40 2022] DELETE /identity/v3/projects/682b9d24e2ac4815bff836ca51ef3780 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:40.973617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0a264a-5cb3-4f46-bf74-b2e2757f7d20 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:40.973899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0a264a-5cb3-4f46-bf74-b2e2757f7d20 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:40.974167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0a264a-5cb3-4f46-bf74-b2e2757f7d20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:40.982496 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:40.993363 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-8a0a264a-5cb3-4f46-bf74-b2e2757f7d20 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:40.994468 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-8a0a264a-5cb3-4f46-bf74-b2e2757f7d20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:41.053086 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1400/2807] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 21 01:37:41.060880 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d68a13-56fc-49eb-a886-4dd74028a8ac None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:41.061158 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d68a13-56fc-49eb-a886-4dd74028a8ac None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:41.061425 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d68a13-56fc-49eb-a886-4dd74028a8ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:41.081298 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-e1d68a13-56fc-49eb-a886-4dd74028a8ac None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:41.083077 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e1d68a13-56fc-49eb-a886-4dd74028a8ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:41.091226 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1408/2808] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:41.095479 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0a264a-5cb3-4f46-bf74-b2e2757f7d20 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:41.105750 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0a264a-5cb3-4f46-bf74-b2e2757f7d20 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:41.110308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b923cbe8-46d4-499d-bf1e-bd3be9d5dec2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:41.110903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b923cbe8-46d4-499d-bf1e-bd3be9d5dec2 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:41.111180 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b923cbe8-46d4-499d-bf1e-bd3be9d5dec2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:41.111448 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b923cbe8-46d4-499d-bf1e-bd3be9d5dec2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:41.112080 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b923cbe8-46d4-499d-bf1e-bd3be9d5dec2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:37:41.112413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b923cbe8-46d4-499d-bf1e-bd3be9d5dec2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:41.114022 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b923cbe8-46d4-499d-bf1e-bd3be9d5dec2 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:41.125806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1401/2809] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:37:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:37:41.130606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d68a13-56fc-49eb-a886-4dd74028a8ac None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:41.139806 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d68a13-56fc-49eb-a886-4dd74028a8ac None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:41.145150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6657e5-cc77-4cc4-a80b-461aa2bcea47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:41.145740 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6657e5-cc77-4cc4-a80b-461aa2bcea47 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:41.146013 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6657e5-cc77-4cc4-a80b-461aa2bcea47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:41.146281 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6657e5-cc77-4cc4-a80b-461aa2bcea47 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:41.147040 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6657e5-cc77-4cc4-a80b-461aa2bcea47 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:41.148940 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6657e5-cc77-4cc4-a80b-461aa2bcea47 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:41.228886 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1409/2810] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:37:41 2022] POST /identity/v3/projects => generated 383 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:41.233233 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b923cbe8-46d4-499d-bf1e-bd3be9d5dec2 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:41.243241 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b923cbe8-46d4-499d-bf1e-bd3be9d5dec2 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:41.300240 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9c2dc7-b013-4e51-86e0-65c364f4ac9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:41.300765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9c2dc7-b013-4e51-86e0-65c364f4ac9b admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:41.301069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9c2dc7-b013-4e51-86e0-65c364f4ac9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:41.301338 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9c2dc7-b013-4e51-86e0-65c364f4ac9b admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:41.301957 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9c2dc7-b013-4e51-86e0-65c364f4ac9b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:41.303853 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9c2dc7-b013-4e51-86e0-65c364f4ac9b admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:41.383195 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1402/2811] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:37:41 2022] POST /identity/v3/users => generated 292 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:41.387346 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6657e5-cc77-4cc4-a80b-461aa2bcea47 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:41.397839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6657e5-cc77-4cc4-a80b-461aa2bcea47 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:41.402413 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec5f55d-eeef-454f-9ae5-6300849f82fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:41.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec5f55d-eeef-454f-9ae5-6300849f82fd admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:41.403274 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec5f55d-eeef-454f-9ae5-6300849f82fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:41.403652 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec5f55d-eeef-454f-9ae5-6300849f82fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:41.404354 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec5f55d-eeef-454f-9ae5-6300849f82fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:41.405920 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec5f55d-eeef-454f-9ae5-6300849f82fd admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:41.415892 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1410/2812] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:41 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:37:41.419860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9c2dc7-b013-4e51-86e0-65c364f4ac9b admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:41.430320 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9c2dc7-b013-4e51-86e0-65c364f4ac9b admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:41.435044 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c523fe-0b1c-4964-8c25-a3aedd20315f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:41.435694 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c523fe-0b1c-4964-8c25-a3aedd20315f admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:41.435967 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c523fe-0b1c-4964-8c25-a3aedd20315f admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:41.436273 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c523fe-0b1c-4964-8c25-a3aedd20315f admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:41.436907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c523fe-0b1c-4964-8c25-a3aedd20315f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:41.438540 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c523fe-0b1c-4964-8c25-a3aedd20315f admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:41.450381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1403/2813] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:37:41.454672 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec5f55d-eeef-454f-9ae5-6300849f82fd admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:41.464931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec5f55d-eeef-454f-9ae5-6300849f82fd admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:41.469508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530681fd-af38-4969-87be-70c6d8555ef4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:41.470145 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530681fd-af38-4969-87be-70c6d8555ef4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:41.470418 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530681fd-af38-4969-87be-70c6d8555ef4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:41.470688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530681fd-af38-4969-87be-70c6d8555ef4 admin admin] PATH_INFO: `/v3/projects/825ee0efa13641b589e8f936fc11f2cc/users/a0ff5633c4b64290b6366b4f0e2c1c94/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:41.485286 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530681fd-af38-4969-87be-70c6d8555ef4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=825ee0efa13641b589e8f936fc11f2cc, user_id=a0ff5633c4b64290b6366b4f0e2c1c94, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:41.487898 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530681fd-af38-4969-87be-70c6d8555ef4 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:41.541874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1411/2814] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:37:41 2022] PUT /identity/v3/projects/825ee0efa13641b589e8f936fc11f2cc/users/a0ff5633c4b64290b6366b4f0e2c1c94/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:41.554020 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c478a5-6243-47df-a251-58d112754676 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:41.554229 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c478a5-6243-47df-a251-58d112754676 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:41.554494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c478a5-6243-47df-a251-58d112754676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:41.581766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-b6c478a5-6243-47df-a251-58d112754676 None None] MFA Rules not processed for user `a0ff5633c4b64290b6366b4f0e2c1c94`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:41.583055 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-b6c478a5-6243-47df-a251-58d112754676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:41.641828 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:41.645425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1404/2815] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:41 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:41.658279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530681fd-af38-4969-87be-70c6d8555ef4 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:41.721037 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530681fd-af38-4969-87be-70c6d8555ef4 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:41.726363 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8940a8a-9f42-439d-b793-3e8bc9376713 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:41.726976 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8940a8a-9f42-439d-b793-3e8bc9376713 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:41.727252 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8940a8a-9f42-439d-b793-3e8bc9376713 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:41.727524 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8940a8a-9f42-439d-b793-3e8bc9376713 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:41.729597 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8940a8a-9f42-439d-b793-3e8bc9376713 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:41.730163 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8940a8a-9f42-439d-b793-3e8bc9376713 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:41.817113 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1412/2816] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:37:41 2022] POST /identity/v3/projects => generated 383 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 21 01:37:41.821079 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c478a5-6243-47df-a251-58d112754676 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:41.831811 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c478a5-6243-47df-a251-58d112754676 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:41.892996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0643be-2876-42ad-b461-08abd82819b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:41.893592 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0643be-2876-42ad-b461-08abd82819b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:41.893866 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0643be-2876-42ad-b461-08abd82819b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:41.894137 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0643be-2876-42ad-b461-08abd82819b7 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:41.894751 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0643be-2876-42ad-b461-08abd82819b7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:41.896672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0643be-2876-42ad-b461-08abd82819b7 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:41.970424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1405/2817] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:37:41 2022] POST /identity/v3/users => generated 298 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:41.974938 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8940a8a-9f42-439d-b793-3e8bc9376713 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:41.985432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8940a8a-9f42-439d-b793-3e8bc9376713 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:41.990692 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75bd9bf-b09f-45d9-9e59-2c3972f03388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:41.991276 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75bd9bf-b09f-45d9-9e59-2c3972f03388 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:41.991547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75bd9bf-b09f-45d9-9e59-2c3972f03388 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:41.991840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75bd9bf-b09f-45d9-9e59-2c3972f03388 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:41.992538 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75bd9bf-b09f-45d9-9e59-2c3972f03388 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:41.994136 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75bd9bf-b09f-45d9-9e59-2c3972f03388 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:42.004440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1413/2818] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:37:42.008556 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0643be-2876-42ad-b461-08abd82819b7 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:42.018685 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0643be-2876-42ad-b461-08abd82819b7 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:42.024792 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635baaa4-dfdb-40a4-87fd-f3fb57c40a00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:42.025436 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635baaa4-dfdb-40a4-87fd-f3fb57c40a00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:42.025699 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635baaa4-dfdb-40a4-87fd-f3fb57c40a00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:42.025965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635baaa4-dfdb-40a4-87fd-f3fb57c40a00 admin admin] PATH_INFO: `/v3/projects/c1d5adfe63fd4de9b35284bcaf99ea97/users/2264be36e42e48dd8c805e8e8e50b1a0/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:42.041329 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635baaa4-dfdb-40a4-87fd-f3fb57c40a00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1d5adfe63fd4de9b35284bcaf99ea97, user_id=2264be36e42e48dd8c805e8e8e50b1a0, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:42.043954 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635baaa4-dfdb-40a4-87fd-f3fb57c40a00 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:42.103232 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1406/2819] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:37:42 2022] PUT /identity/v3/projects/c1d5adfe63fd4de9b35284bcaf99ea97/users/2264be36e42e48dd8c805e8e8e50b1a0/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:42.107850 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75bd9bf-b09f-45d9-9e59-2c3972f03388 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:42.121084 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75bd9bf-b09f-45d9-9e59-2c3972f03388 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:42.125785 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:42.176982 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465826a1-c885-41bc-8b70-87e24cf82e08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:42.177580 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465826a1-c885-41bc-8b70-87e24cf82e08 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:42.177854 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465826a1-c885-41bc-8b70-87e24cf82e08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:42.178122 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465826a1-c885-41bc-8b70-87e24cf82e08 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:42.178745 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465826a1-c885-41bc-8b70-87e24cf82e08 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:42.180348 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465826a1-c885-41bc-8b70-87e24cf82e08 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:42.190078 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1414/2820] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:42 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:37:42.194339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635baaa4-dfdb-40a4-87fd-f3fb57c40a00 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:42.204969 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635baaa4-dfdb-40a4-87fd-f3fb57c40a00 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:42.210594 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9831c0e6-d15b-4ca2-8a7b-4b0539197f21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:42.211234 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9831c0e6-d15b-4ca2-8a7b-4b0539197f21 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:42.211512 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9831c0e6-d15b-4ca2-8a7b-4b0539197f21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:42.211788 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9831c0e6-d15b-4ca2-8a7b-4b0539197f21 admin admin] PATH_INFO: `/v3/projects/c1d5adfe63fd4de9b35284bcaf99ea97/users/2264be36e42e48dd8c805e8e8e50b1a0/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:42.213762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9831c0e6-d15b-4ca2-8a7b-4b0539197f21 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1d5adfe63fd4de9b35284bcaf99ea97, user_id=2264be36e42e48dd8c805e8e8e50b1a0, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:42.216476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9831c0e6-d15b-4ca2-8a7b-4b0539197f21 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:42.227453 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1407/2821] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:37:42 2022] PUT /identity/v3/projects/c1d5adfe63fd4de9b35284bcaf99ea97/users/2264be36e42e48dd8c805e8e8e50b1a0/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:42.237553 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb8c3c1-3923-49da-8209-837decb3b67a None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:42.237836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb8c3c1-3923-49da-8209-837decb3b67a None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:42.238135 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb8c3c1-3923-49da-8209-837decb3b67a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:42.265796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-adb8c3c1-3923-49da-8209-837decb3b67a None None] MFA Rules not processed for user `2264be36e42e48dd8c805e8e8e50b1a0`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:42.267009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-adb8c3c1-3923-49da-8209-837decb3b67a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:42.335584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1415/2822] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:42 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:42.346184 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9831c0e6-d15b-4ca2-8a7b-4b0539197f21 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:42.350268 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-9831c0e6-d15b-4ca2-8a7b-4b0539197f21 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:42.407761 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9831c0e6-d15b-4ca2-8a7b-4b0539197f21 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:42.412281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fd93e4-267f-4070-bf57-87ed59d433f0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:42.412741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fd93e4-267f-4070-bf57-87ed59d433f0 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:42.413015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fd93e4-267f-4070-bf57-87ed59d433f0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:42.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fd93e4-267f-4070-bf57-87ed59d433f0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:42.422181 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fd93e4-267f-4070-bf57-87ed59d433f0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:42.423641 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fd93e4-267f-4070-bf57-87ed59d433f0 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:42.433010 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1408/2823] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:42 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:37:42.536165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb8c3c1-3923-49da-8209-837decb3b67a None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:42.545628 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb8c3c1-3923-49da-8209-837decb3b67a None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:42.549226 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59920722-3d36-45b1-9ec0-4a1c59313988 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:42.549811 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59920722-3d36-45b1-9ec0-4a1c59313988 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:42.550082 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59920722-3d36-45b1-9ec0-4a1c59313988 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:42.550349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59920722-3d36-45b1-9ec0-4a1c59313988 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:42.606287 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59920722-3d36-45b1-9ec0-4a1c59313988 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:42.607698 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59920722-3d36-45b1-9ec0-4a1c59313988 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:42.616580 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1416/2824] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:42 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:37:45.640942 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fd93e4-267f-4070-bf57-87ed59d433f0 service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:45.699441 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:45.704961 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fd93e4-267f-4070-bf57-87ed59d433f0 service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:45.709899 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5deb65-c7a7-49e5-8750-93f70ac11824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:45.711801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5deb65-c7a7-49e5-8750-93f70ac11824 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:45.711801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5deb65-c7a7-49e5-8750-93f70ac11824 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:45.711801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5deb65-c7a7-49e5-8750-93f70ac11824 admin admin] PATH_INFO: `/v3/users/a0ff5633c4b64290b6366b4f0e2c1c94` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:45.712249 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5deb65-c7a7-49e5-8750-93f70ac11824 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a0ff5633c4b64290b6366b4f0e2c1c94)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:45.714167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5deb65-c7a7-49e5-8750-93f70ac11824 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:45.961894 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-bb5deb65-c7a7-49e5-8750-93f70ac11824 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a0ff5633c4b64290b6366b4f0e2c1c94'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:45.968909 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-bb5deb65-c7a7-49e5-8750-93f70ac11824 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a0ff5633c4b64290b6366b4f0e2c1c94'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:45.969622 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-bb5deb65-c7a7-49e5-8750-93f70ac11824 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a0ff5633c4b64290b6366b4f0e2c1c94'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:45.982318 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-bb5deb65-c7a7-49e5-8750-93f70ac11824 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a0ff5633c4b64290b6366b4f0e2c1c94'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:46.073191 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-bb5deb65-c7a7-49e5-8750-93f70ac11824 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a0ff5633c4b64290b6366b4f0e2c1c94'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:46.079497 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1409/2825] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:37:45 2022] DELETE /identity/v3/users/a0ff5633c4b64290b6366b4f0e2c1c94 => generated 0 bytes in 439 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:37:46.084960 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eca4a29-86c5-4162-91eb-ba19a1ff60d1 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:46.085939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eca4a29-86c5-4162-91eb-ba19a1ff60d1 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:46.085939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eca4a29-86c5-4162-91eb-ba19a1ff60d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:46.104907 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-6eca4a29-86c5-4162-91eb-ba19a1ff60d1 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:46.106071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6eca4a29-86c5-4162-91eb-ba19a1ff60d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:46.128986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=100 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:46.163556 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1417/2826] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:46.173393 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5deb65-c7a7-49e5-8750-93f70ac11824 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:46.174815 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-bb5deb65-c7a7-49e5-8750-93f70ac11824 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:46.233564 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5deb65-c7a7-49e5-8750-93f70ac11824 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:46.237185 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba27d4b2-adc1-4c7e-9ec2-8f00e680d2a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:46.237782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba27d4b2-adc1-4c7e-9ec2-8f00e680d2a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:46.238055 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba27d4b2-adc1-4c7e-9ec2-8f00e680d2a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:46.238321 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba27d4b2-adc1-4c7e-9ec2-8f00e680d2a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:46.247281 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba27d4b2-adc1-4c7e-9ec2-8f00e680d2a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:46.249364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba27d4b2-adc1-4c7e-9ec2-8f00e680d2a0 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:46.261116 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1410/2827] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:37:46.652162 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eca4a29-86c5-4162-91eb-ba19a1ff60d1 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:46.653670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-6eca4a29-86c5-4162-91eb-ba19a1ff60d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:46.666083 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eca4a29-86c5-4162-91eb-ba19a1ff60d1 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:46.671177 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:46.671815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:46.672087 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:46.672384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5 admin admin] PATH_INFO: `/v3/projects/825ee0efa13641b589e8f936fc11f2cc` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:46.674273 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=825ee0efa13641b589e8f936fc11f2cc)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:46.681440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:46.776050 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '825ee0efa13641b589e8f936fc11f2cc', 'request_id': 'req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:46.827353 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '825ee0efa13641b589e8f936fc11f2cc', 'request_id': 'req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:46.833339 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '825ee0efa13641b589e8f936fc11f2cc', 'request_id': 'req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:46.845221 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5 admin admin] The token cache is being invalidate because project 825ee0efa13641b589e8f936fc11f2cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:37:46.845509 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:46.846225 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:46.847769 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1418/2828] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:37:46 2022] DELETE /identity/v3/projects/825ee0efa13641b589e8f936fc11f2cc => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:46.851794 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba27d4b2-adc1-4c7e-9ec2-8f00e680d2a0 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:46.864627 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba27d4b2-adc1-4c7e-9ec2-8f00e680d2a0 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:46.918270 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551e3925-737c-4849-bc5f-d9099564fda9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:46.918929 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551e3925-737c-4849-bc5f-d9099564fda9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:46.919208 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551e3925-737c-4849-bc5f-d9099564fda9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:46.919481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551e3925-737c-4849-bc5f-d9099564fda9 admin admin] PATH_INFO: `/v3/users/2264be36e42e48dd8c805e8e8e50b1a0` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:46.920622 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551e3925-737c-4849-bc5f-d9099564fda9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2264be36e42e48dd8c805e8e8e50b1a0)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:46.922962 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551e3925-737c-4849-bc5f-d9099564fda9 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:46.944677 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:47.127500 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-551e3925-737c-4849-bc5f-d9099564fda9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2264be36e42e48dd8c805e8e8e50b1a0'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:47.134907 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-551e3925-737c-4849-bc5f-d9099564fda9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2264be36e42e48dd8c805e8e8e50b1a0'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:47.135572 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-551e3925-737c-4849-bc5f-d9099564fda9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2264be36e42e48dd8c805e8e8e50b1a0'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:47.147650 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-551e3925-737c-4849-bc5f-d9099564fda9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2264be36e42e48dd8c805e8e8e50b1a0'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:47.215790 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-551e3925-737c-4849-bc5f-d9099564fda9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2264be36e42e48dd8c805e8e8e50b1a0'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:47.223070 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1411/2829] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:37:46 2022] DELETE /identity/v3/users/2264be36e42e48dd8c805e8e8e50b1a0 => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:47.607148 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:47.608629 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:47.631309 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2a9fa4-7ae9-4f43-b74c-e1806c37cee5 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:47.689334 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2306d021-2bae-497b-8703-8c0e198a8227 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:47.689865 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2306d021-2bae-497b-8703-8c0e198a8227 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:47.690139 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2306d021-2bae-497b-8703-8c0e198a8227 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:47.690410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2306d021-2bae-497b-8703-8c0e198a8227 admin admin] PATH_INFO: `/v3/projects/c1d5adfe63fd4de9b35284bcaf99ea97` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:47.691521 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2306d021-2bae-497b-8703-8c0e198a8227 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c1d5adfe63fd4de9b35284bcaf99ea97)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:47.693933 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2306d021-2bae-497b-8703-8c0e198a8227 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:47.837673 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2306d021-2bae-497b-8703-8c0e198a8227 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c1d5adfe63fd4de9b35284bcaf99ea97', 'request_id': 'req-2306d021-2bae-497b-8703-8c0e198a8227'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:47.841807 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:47.896410 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2306d021-2bae-497b-8703-8c0e198a8227 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c1d5adfe63fd4de9b35284bcaf99ea97', 'request_id': 'req-2306d021-2bae-497b-8703-8c0e198a8227'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:47.902163 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2306d021-2bae-497b-8703-8c0e198a8227 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c1d5adfe63fd4de9b35284bcaf99ea97', 'request_id': 'req-2306d021-2bae-497b-8703-8c0e198a8227'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:47.913606 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2306d021-2bae-497b-8703-8c0e198a8227 admin admin] The token cache is being invalidate because project c1d5adfe63fd4de9b35284bcaf99ea97 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:37:47.917003 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2306d021-2bae-497b-8703-8c0e198a8227 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:47.917003 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-2306d021-2bae-497b-8703-8c0e198a8227 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:47.917003 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1419/2830] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:37:47 2022] DELETE /identity/v3/projects/c1d5adfe63fd4de9b35284bcaf99ea97 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:47.921465 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfe48da-0714-4de4-856e-566e119b6bc3 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:47.921965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfe48da-0714-4de4-856e-566e119b6bc3 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:47.922076 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfe48da-0714-4de4-856e-566e119b6bc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:47.943703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-ecfe48da-0714-4de4-856e-566e119b6bc3 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:47.943703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-ecfe48da-0714-4de4-856e-566e119b6bc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:48.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1412/2831] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:48.011343 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c924120-f421-4a03-a341-a905ea7d3f76 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:48.011623 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c924120-f421-4a03-a341-a905ea7d3f76 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:48.011893 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c924120-f421-4a03-a341-a905ea7d3f76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:48.031839 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-4c924120-f421-4a03-a341-a905ea7d3f76 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:48.033072 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-4c924120-f421-4a03-a341-a905ea7d3f76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:48.041016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1420/2832] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:48.045401 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfe48da-0714-4de4-856e-566e119b6bc3 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:48.055739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfe48da-0714-4de4-856e-566e119b6bc3 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:48.060381 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef93a9b4-80cd-433d-9899-895d01a65d23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:48.060968 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef93a9b4-80cd-433d-9899-895d01a65d23 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:48.061241 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef93a9b4-80cd-433d-9899-895d01a65d23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:48.061505 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef93a9b4-80cd-433d-9899-895d01a65d23 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:48.062159 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef93a9b4-80cd-433d-9899-895d01a65d23 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:37:48.062468 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef93a9b4-80cd-433d-9899-895d01a65d23 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:48.064015 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef93a9b4-80cd-433d-9899-895d01a65d23 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:48.076494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1413/2833] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:37:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:37:48.080904 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c924120-f421-4a03-a341-a905ea7d3f76 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:48.090250 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c924120-f421-4a03-a341-a905ea7d3f76 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:48.094918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9287b34e-a116-4e98-a739-b10ba795bafe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:48.095512 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9287b34e-a116-4e98-a739-b10ba795bafe admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:48.095786 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9287b34e-a116-4e98-a739-b10ba795bafe admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:48.096053 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9287b34e-a116-4e98-a739-b10ba795bafe admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:48.096702 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9287b34e-a116-4e98-a739-b10ba795bafe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:48.098584 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9287b34e-a116-4e98-a739-b10ba795bafe admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:48.179793 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1421/2834] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:37:48 2022] POST /identity/v3/projects => generated 377 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:48.184131 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef93a9b4-80cd-433d-9899-895d01a65d23 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:48.194998 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef93a9b4-80cd-433d-9899-895d01a65d23 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:48.251707 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1e3333-5a98-4db0-a4c5-596baef205de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:48.252364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1e3333-5a98-4db0-a4c5-596baef205de admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:48.252642 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1e3333-5a98-4db0-a4c5-596baef205de admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:48.252910 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1e3333-5a98-4db0-a4c5-596baef205de admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:48.253529 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1e3333-5a98-4db0-a4c5-596baef205de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:48.256047 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1e3333-5a98-4db0-a4c5-596baef205de admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:48.336445 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1414/2835] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:37:48 2022] POST /identity/v3/users => generated 289 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:48.341363 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9287b34e-a116-4e98-a739-b10ba795bafe admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:48.358797 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9287b34e-a116-4e98-a739-b10ba795bafe admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:48.366702 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69309cc6-00c6-4260-b65b-101a5c497de2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:48.367681 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69309cc6-00c6-4260-b65b-101a5c497de2 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:48.368144 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69309cc6-00c6-4260-b65b-101a5c497de2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:48.368582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69309cc6-00c6-4260-b65b-101a5c497de2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:48.369662 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69309cc6-00c6-4260-b65b-101a5c497de2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:48.372542 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69309cc6-00c6-4260-b65b-101a5c497de2 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:48.391119 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1422/2836] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:48 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:37:48.395296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1e3333-5a98-4db0-a4c5-596baef205de admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:48.405172 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1e3333-5a98-4db0-a4c5-596baef205de admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:48.409813 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c903ff-c7ba-4bd5-9e49-641e9d6bd37a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:48.410409 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c903ff-c7ba-4bd5-9e49-641e9d6bd37a admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:48.410690 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c903ff-c7ba-4bd5-9e49-641e9d6bd37a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:48.411447 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c903ff-c7ba-4bd5-9e49-641e9d6bd37a admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:48.412074 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c903ff-c7ba-4bd5-9e49-641e9d6bd37a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:48.413676 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c903ff-c7ba-4bd5-9e49-641e9d6bd37a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:48.424136 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1415/2837] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:48 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:37:48.428908 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69309cc6-00c6-4260-b65b-101a5c497de2 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:48.438630 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69309cc6-00c6-4260-b65b-101a5c497de2 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:48.443182 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c55846-347c-4ecf-8621-f2e4b78af46e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:48.443814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c55846-347c-4ecf-8621-f2e4b78af46e admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:48.444087 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c55846-347c-4ecf-8621-f2e4b78af46e admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:48.444383 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c55846-347c-4ecf-8621-f2e4b78af46e admin admin] PATH_INFO: `/v3/projects/027ee8725121430391fc46db1e07ac80/users/df7c011c7d10460e8002e91905bc295c/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:48.458796 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c55846-347c-4ecf-8621-f2e4b78af46e admin admin] RBAC: Authorizing `identity:create_grant(project_id=027ee8725121430391fc46db1e07ac80, user_id=df7c011c7d10460e8002e91905bc295c, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:48.461717 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c55846-347c-4ecf-8621-f2e4b78af46e admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:48.537009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1423/2838] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:37:48 2022] PUT /identity/v3/projects/027ee8725121430391fc46db1e07ac80/users/df7c011c7d10460e8002e91905bc295c/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:37:48.547468 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bc9bd6-683a-4d84-8890-c98498513f17 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:48.547739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bc9bd6-683a-4d84-8890-c98498513f17 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:48.548007 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bc9bd6-683a-4d84-8890-c98498513f17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:48.555785 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:DELETE=8,keystone:SELECT=88,keystone:INSERT=4 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:48.576500 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-47bc9bd6-683a-4d84-8890-c98498513f17 None None] MFA Rules not processed for user `df7c011c7d10460e8002e91905bc295c`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:48.577708 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-47bc9bd6-683a-4d84-8890-c98498513f17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:48.641683 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1416/2839] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:48 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:48.646291 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c55846-347c-4ecf-8621-f2e4b78af46e admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:48.709862 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c55846-347c-4ecf-8621-f2e4b78af46e admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:48.714700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7593e6-e196-42f5-96c4-d584407206ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:48.715299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7593e6-e196-42f5-96c4-d584407206ef admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:48.715570 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7593e6-e196-42f5-96c4-d584407206ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:48.715864 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7593e6-e196-42f5-96c4-d584407206ef admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:48.716506 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7593e6-e196-42f5-96c4-d584407206ef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:48.718451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7593e6-e196-42f5-96c4-d584407206ef admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:48.788462 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1424/2840] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:37:48 2022] POST /identity/v3/projects => generated 379 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:48.792763 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bc9bd6-683a-4d84-8890-c98498513f17 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:48.804609 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bc9bd6-683a-4d84-8890-c98498513f17 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:48.861042 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50de94a7-b8f5-4794-a657-bb6807bfe3cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:48.861647 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50de94a7-b8f5-4794-a657-bb6807bfe3cf admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:48.861925 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50de94a7-b8f5-4794-a657-bb6807bfe3cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:48.862193 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50de94a7-b8f5-4794-a657-bb6807bfe3cf admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:48.862822 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50de94a7-b8f5-4794-a657-bb6807bfe3cf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:48.865326 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50de94a7-b8f5-4794-a657-bb6807bfe3cf admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:48.941562 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1417/2841] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:37:48 2022] POST /identity/v3/users => generated 290 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:48.946422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7593e6-e196-42f5-96c4-d584407206ef admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:48.956950 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7593e6-e196-42f5-96c4-d584407206ef admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:48.961939 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e1ddc0-a14b-4f22-9459-474aa7a84df0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:48.962479 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e1ddc0-a14b-4f22-9459-474aa7a84df0 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:48.962755 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e1ddc0-a14b-4f22-9459-474aa7a84df0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:48.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e1ddc0-a14b-4f22-9459-474aa7a84df0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:48.963660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e1ddc0-a14b-4f22-9459-474aa7a84df0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:48.965270 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e1ddc0-a14b-4f22-9459-474aa7a84df0 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:48.975827 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1425/2842] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:48 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:37:48.980197 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50de94a7-b8f5-4794-a657-bb6807bfe3cf admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:48.993621 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50de94a7-b8f5-4794-a657-bb6807bfe3cf admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:48.998367 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a630169d-e560-4239-adf2-7bf77ffb81c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:48.998951 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a630169d-e560-4239-adf2-7bf77ffb81c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:48.999228 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a630169d-e560-4239-adf2-7bf77ffb81c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:48.999495 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a630169d-e560-4239-adf2-7bf77ffb81c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:49.000136 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a630169d-e560-4239-adf2-7bf77ffb81c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:49.001710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a630169d-e560-4239-adf2-7bf77ffb81c7 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:49.012269 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1418/2843] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:48 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:37:49.016546 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e1ddc0-a14b-4f22-9459-474aa7a84df0 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:49.026744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e1ddc0-a14b-4f22-9459-474aa7a84df0 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:49.031716 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3684ef11-718e-492a-8ebf-825a5707c230 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:49.032385 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3684ef11-718e-492a-8ebf-825a5707c230 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:49.032660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3684ef11-718e-492a-8ebf-825a5707c230 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:49.032930 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3684ef11-718e-492a-8ebf-825a5707c230 admin admin] PATH_INFO: `/v3/projects/79aa769b2bcd4f9c8a6931df8ef6a398/users/17450ef1dfd740c3b44a4db9e5bcf30b/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:49.047169 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3684ef11-718e-492a-8ebf-825a5707c230 admin admin] RBAC: Authorizing `identity:create_grant(project_id=79aa769b2bcd4f9c8a6931df8ef6a398, user_id=17450ef1dfd740c3b44a4db9e5bcf30b, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:49.049838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3684ef11-718e-492a-8ebf-825a5707c230 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:49.118953 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1426/2844] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:37:49 2022] PUT /identity/v3/projects/79aa769b2bcd4f9c8a6931df8ef6a398/users/17450ef1dfd740c3b44a4db9e5bcf30b/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:49.130203 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbd537f-2816-4047-ba41-8b43978e1b98 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:49.130492 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbd537f-2816-4047-ba41-8b43978e1b98 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:49.130762 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbd537f-2816-4047-ba41-8b43978e1b98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:49.162494 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-5cbd537f-2816-4047-ba41-8b43978e1b98 None None] MFA Rules not processed for user `17450ef1dfd740c3b44a4db9e5bcf30b`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:49.163710 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-5cbd537f-2816-4047-ba41-8b43978e1b98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:49.224242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:49.227756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1419/2845] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:49 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:49.232326 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3684ef11-718e-492a-8ebf-825a5707c230 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:49.265763 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:49.295823 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3684ef11-718e-492a-8ebf-825a5707c230 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:49.300511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2c254f-af1f-414a-b278-d675402d1f38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:49.301110 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2c254f-af1f-414a-b278-d675402d1f38 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:49.301387 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2c254f-af1f-414a-b278-d675402d1f38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:49.302198 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2c254f-af1f-414a-b278-d675402d1f38 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:49.302818 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2c254f-af1f-414a-b278-d675402d1f38 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:49.304733 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2c254f-af1f-414a-b278-d675402d1f38 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:49.389981 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1427/2846] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:37:49 2022] POST /identity/v3/projects => generated 379 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:49.394135 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbd537f-2816-4047-ba41-8b43978e1b98 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:49.405019 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbd537f-2816-4047-ba41-8b43978e1b98 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:49.462093 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ce9d1e-efc7-4818-afc0-1b9e018ec8e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:49.462756 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ce9d1e-efc7-4818-afc0-1b9e018ec8e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:49.463029 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ce9d1e-efc7-4818-afc0-1b9e018ec8e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:49.463296 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ce9d1e-efc7-4818-afc0-1b9e018ec8e5 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:49.463939 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ce9d1e-efc7-4818-afc0-1b9e018ec8e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:49.465864 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ce9d1e-efc7-4818-afc0-1b9e018ec8e5 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:49.518960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1420/2847] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:37:49 2022] POST /identity/v3/users => generated 296 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:49.523060 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2c254f-af1f-414a-b278-d675402d1f38 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:49.533451 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2c254f-af1f-414a-b278-d675402d1f38 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:49.539067 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ec76ca-3659-43cd-8795-331d443b8adc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:49.539673 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ec76ca-3659-43cd-8795-331d443b8adc admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:49.539948 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ec76ca-3659-43cd-8795-331d443b8adc admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:49.540257 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ec76ca-3659-43cd-8795-331d443b8adc admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:49.540931 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ec76ca-3659-43cd-8795-331d443b8adc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:49.542511 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ec76ca-3659-43cd-8795-331d443b8adc admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:49.553523 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1428/2848] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:49 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:37:49.558013 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ce9d1e-efc7-4818-afc0-1b9e018ec8e5 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:49.568786 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ce9d1e-efc7-4818-afc0-1b9e018ec8e5 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:49.573613 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30062d8-a702-49ea-b55a-df4b8b8345f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:49.574263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30062d8-a702-49ea-b55a-df4b8b8345f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:49.574626 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30062d8-a702-49ea-b55a-df4b8b8345f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:49.574965 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30062d8-a702-49ea-b55a-df4b8b8345f1 admin admin] PATH_INFO: `/v3/projects/5559837564614bc4bc5097d31d97a6b3/users/8ce68660cf854c1dacb60b6a7f37f21b/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:49.590867 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30062d8-a702-49ea-b55a-df4b8b8345f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5559837564614bc4bc5097d31d97a6b3, user_id=8ce68660cf854c1dacb60b6a7f37f21b, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:49.593666 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30062d8-a702-49ea-b55a-df4b8b8345f1 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:49.673622 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1421/2849] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:37:49 2022] PUT /identity/v3/projects/5559837564614bc4bc5097d31d97a6b3/users/8ce68660cf854c1dacb60b6a7f37f21b/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:49.678384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ec76ca-3659-43cd-8795-331d443b8adc admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:49.688819 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ec76ca-3659-43cd-8795-331d443b8adc admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:49.745835 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273eafa6-28e0-407c-b385-f6afdb28ae7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:49.746432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273eafa6-28e0-407c-b385-f6afdb28ae7a admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:49.746702 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273eafa6-28e0-407c-b385-f6afdb28ae7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:49.746966 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273eafa6-28e0-407c-b385-f6afdb28ae7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:49.747589 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273eafa6-28e0-407c-b385-f6afdb28ae7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:49.749181 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273eafa6-28e0-407c-b385-f6afdb28ae7a admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:49.759224 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1429/2850] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:49 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:37:49.763463 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30062d8-a702-49ea-b55a-df4b8b8345f1 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:49.774745 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30062d8-a702-49ea-b55a-df4b8b8345f1 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:49.779360 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b83c95-5792-4231-8ccf-23a03214bf74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:49.780010 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b83c95-5792-4231-8ccf-23a03214bf74 admin admin] REQUEST_METHOD: `PUT` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:49.780444 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b83c95-5792-4231-8ccf-23a03214bf74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:49.780636 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b83c95-5792-4231-8ccf-23a03214bf74 admin admin] PATH_INFO: `/v3/projects/5559837564614bc4bc5097d31d97a6b3/users/8ce68660cf854c1dacb60b6a7f37f21b/roles/ea4472fe9a96464888f3eb593eb46674` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:49.782546 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b83c95-5792-4231-8ccf-23a03214bf74 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5559837564614bc4bc5097d31d97a6b3, user_id=8ce68660cf854c1dacb60b6a7f37f21b, role_id=ea4472fe9a96464888f3eb593eb46674)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:49.785210 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b83c95-5792-4231-8ccf-23a03214bf74 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:49.796205 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1422/2851] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:37:49 2022] PUT /identity/v3/projects/5559837564614bc4bc5097d31d97a6b3/users/8ce68660cf854c1dacb60b6a7f37f21b/roles/ea4472fe9a96464888f3eb593eb46674 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:49.807475 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324f8b79-9eb8-4e5b-9592-a04dffb01c7b None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:49.807758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324f8b79-9eb8-4e5b-9592-a04dffb01c7b None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:49.808029 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324f8b79-9eb8-4e5b-9592-a04dffb01c7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:49.836449 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-324f8b79-9eb8-4e5b-9592-a04dffb01c7b None None] MFA Rules not processed for user `8ce68660cf854c1dacb60b6a7f37f21b`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:49.837718 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-324f8b79-9eb8-4e5b-9592-a04dffb01c7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:49.905121 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1430/2852] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:49 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:49.916988 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b83c95-5792-4231-8ccf-23a03214bf74 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:49.918714 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d4b83c95-5792-4231-8ccf-23a03214bf74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:49.980374 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b83c95-5792-4231-8ccf-23a03214bf74 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:49.984157 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f361ce4-4c4d-4de5-b680-92c26916b0c0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:49.984766 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f361ce4-4c4d-4de5-b680-92c26916b0c0 service cinder] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:49.985058 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f361ce4-4c4d-4de5-b680-92c26916b0c0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:49.985327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f361ce4-4c4d-4de5-b680-92c26916b0c0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:49.994359 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f361ce4-4c4d-4de5-b680-92c26916b0c0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:49.995778 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f361ce4-4c4d-4de5-b680-92c26916b0c0 service cinder] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:50.005650 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1423/2853] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:49 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:37:50.100899 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324f8b79-9eb8-4e5b-9592-a04dffb01c7b None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:50.102583 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-324f8b79-9eb8-4e5b-9592-a04dffb01c7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:50.115284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324f8b79-9eb8-4e5b-9592-a04dffb01c7b None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:50.119209 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fd287b-d4d6-44a9-9604-d34384b97029 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:50.119853 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fd287b-d4d6-44a9-9604-d34384b97029 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:50.120150 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fd287b-d4d6-44a9-9604-d34384b97029 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:50.120466 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fd287b-d4d6-44a9-9604-d34384b97029 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:50.142478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:50.185918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fd287b-d4d6-44a9-9604-d34384b97029 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:50.187361 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fd287b-d4d6-44a9-9604-d34384b97029 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:50.197383 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1431/2854] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:50 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:37:54.224764 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f361ce4-4c4d-4de5-b680-92c26916b0c0 service cinder] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:54.283876 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:54.289473 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f361ce4-4c4d-4de5-b680-92c26916b0c0 service cinder] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:54.294242 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574becb1-c95e-4de6-9292-1314358b5c13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:54.294869 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574becb1-c95e-4de6-9292-1314358b5c13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:54.299375 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574becb1-c95e-4de6-9292-1314358b5c13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:54.299375 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574becb1-c95e-4de6-9292-1314358b5c13 admin admin] PATH_INFO: `/v3/users/df7c011c7d10460e8002e91905bc295c` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:54.299375 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574becb1-c95e-4de6-9292-1314358b5c13 admin admin] RBAC: Authorizing `identity:delete_user(user_id=df7c011c7d10460e8002e91905bc295c)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:54.299375 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574becb1-c95e-4de6-9292-1314358b5c13 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:54.477143 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-574becb1-c95e-4de6-9292-1314358b5c13 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df7c011c7d10460e8002e91905bc295c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:54.485280 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-574becb1-c95e-4de6-9292-1314358b5c13 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df7c011c7d10460e8002e91905bc295c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:54.486123 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-574becb1-c95e-4de6-9292-1314358b5c13 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df7c011c7d10460e8002e91905bc295c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:54.499106 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-574becb1-c95e-4de6-9292-1314358b5c13 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df7c011c7d10460e8002e91905bc295c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:54.581413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-574becb1-c95e-4de6-9292-1314358b5c13 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df7c011c7d10460e8002e91905bc295c'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:54.588043 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1424/2855] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:37:54 2022] DELETE /identity/v3/users/df7c011c7d10460e8002e91905bc295c => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:54.592774 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a68869-db97-4d23-93a6-7aba8a8c0a3b None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:54.593254 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a68869-db97-4d23-93a6-7aba8a8c0a3b None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:54.593357 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a68869-db97-4d23-93a6-7aba8a8c0a3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:54.612547 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-05a68869-db97-4d23-93a6-7aba8a8c0a3b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:54.614277 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-05a68869-db97-4d23-93a6-7aba8a8c0a3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:54.671667 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1432/2856] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:54.682145 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574becb1-c95e-4de6-9292-1314358b5c13 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:54.683626 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-574becb1-c95e-4de6-9292-1314358b5c13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:54.742297 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574becb1-c95e-4de6-9292-1314358b5c13 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:54.745953 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f7b204-4c26-4f52-b68f-16a503f75ca5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:54.746554 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f7b204-4c26-4f52-b68f-16a503f75ca5 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:54.746832 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f7b204-4c26-4f52-b68f-16a503f75ca5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:54.747108 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f7b204-4c26-4f52-b68f-16a503f75ca5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:54.757782 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f7b204-4c26-4f52-b68f-16a503f75ca5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:54.759201 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f7b204-4c26-4f52-b68f-16a503f75ca5 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:54.768424 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1425/2857] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:37:55.161492 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a68869-db97-4d23-93a6-7aba8a8c0a3b None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:55.163091 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-05a68869-db97-4d23-93a6-7aba8a8c0a3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:55.176288 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a68869-db97-4d23-93a6-7aba8a8c0a3b None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:55.181027 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729f0bd5-af98-4268-92e8-d6487b758bba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:55.181688 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729f0bd5-af98-4268-92e8-d6487b758bba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:55.181958 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729f0bd5-af98-4268-92e8-d6487b758bba admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:55.182231 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729f0bd5-af98-4268-92e8-d6487b758bba admin admin] PATH_INFO: `/v3/projects/027ee8725121430391fc46db1e07ac80` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:55.183416 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729f0bd5-af98-4268-92e8-d6487b758bba admin admin] RBAC: Authorizing `identity:delete_project(project_id=027ee8725121430391fc46db1e07ac80)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:55.185414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729f0bd5-af98-4268-92e8-d6487b758bba admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:55.262565 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-729f0bd5-af98-4268-92e8-d6487b758bba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '027ee8725121430391fc46db1e07ac80', 'request_id': 'req-729f0bd5-af98-4268-92e8-d6487b758bba'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:55.313128 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-729f0bd5-af98-4268-92e8-d6487b758bba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '027ee8725121430391fc46db1e07ac80', 'request_id': 'req-729f0bd5-af98-4268-92e8-d6487b758bba'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:55.319672 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-729f0bd5-af98-4268-92e8-d6487b758bba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '027ee8725121430391fc46db1e07ac80', 'request_id': 'req-729f0bd5-af98-4268-92e8-d6487b758bba'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:55.331031 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-729f0bd5-af98-4268-92e8-d6487b758bba admin admin] The token cache is being invalidate because project 027ee8725121430391fc46db1e07ac80 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:37:55.331319 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-729f0bd5-af98-4268-92e8-d6487b758bba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:55.331966 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-729f0bd5-af98-4268-92e8-d6487b758bba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:55.334025 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1433/2858] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:37:55 2022] DELETE /identity/v3/projects/027ee8725121430391fc46db1e07ac80 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:55.338538 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f7b204-4c26-4f52-b68f-16a503f75ca5 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:55.349067 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f7b204-4c26-4f52-b68f-16a503f75ca5 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:55.409025 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18623a87-90aa-4f0d-9b77-82f576701809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:55.409687 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18623a87-90aa-4f0d-9b77-82f576701809 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:55.409964 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18623a87-90aa-4f0d-9b77-82f576701809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:55.410238 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18623a87-90aa-4f0d-9b77-82f576701809 admin admin] PATH_INFO: `/v3/users/17450ef1dfd740c3b44a4db9e5bcf30b` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:55.411425 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18623a87-90aa-4f0d-9b77-82f576701809 admin admin] RBAC: Authorizing `identity:delete_user(user_id=17450ef1dfd740c3b44a4db9e5bcf30b)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:55.413430 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18623a87-90aa-4f0d-9b77-82f576701809 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:55.434327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:55.583229 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-18623a87-90aa-4f0d-9b77-82f576701809 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '17450ef1dfd740c3b44a4db9e5bcf30b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:55.590289 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-18623a87-90aa-4f0d-9b77-82f576701809 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '17450ef1dfd740c3b44a4db9e5bcf30b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:55.590960 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-18623a87-90aa-4f0d-9b77-82f576701809 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '17450ef1dfd740c3b44a4db9e5bcf30b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:55.603422 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-18623a87-90aa-4f0d-9b77-82f576701809 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '17450ef1dfd740c3b44a4db9e5bcf30b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:55.667143 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-18623a87-90aa-4f0d-9b77-82f576701809 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '17450ef1dfd740c3b44a4db9e5bcf30b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:55.673560 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1426/2859] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:37:55 2022] DELETE /identity/v3/users/17450ef1dfd740c3b44a4db9e5bcf30b => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:56.131405 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729f0bd5-af98-4268-92e8-d6487b758bba admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:56.133124 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-729f0bd5-af98-4268-92e8-d6487b758bba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:56.149539 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729f0bd5-af98-4268-92e8-d6487b758bba admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:56.196535 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:56.209090 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:56.217284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:56.217284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:56.217284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4 admin admin] PATH_INFO: `/v3/projects/79aa769b2bcd4f9c8a6931df8ef6a398` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:56.217284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=79aa769b2bcd4f9c8a6931df8ef6a398)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:56.217284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:56.343404 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '79aa769b2bcd4f9c8a6931df8ef6a398', 'request_id': 'req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:56.398388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '79aa769b2bcd4f9c8a6931df8ef6a398', 'request_id': 'req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:56.404414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '79aa769b2bcd4f9c8a6931df8ef6a398', 'request_id': 'req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:56.416736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4 admin admin] The token cache is being invalidate because project 79aa769b2bcd4f9c8a6931df8ef6a398 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:37:56.417039 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:56.417684 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:56.420728 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1434/2860] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:37:56 2022] DELETE /identity/v3/projects/79aa769b2bcd4f9c8a6931df8ef6a398 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:56.423457 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18623a87-90aa-4f0d-9b77-82f576701809 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:56.425005 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-18623a87-90aa-4f0d-9b77-82f576701809 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:56.437744 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18623a87-90aa-4f0d-9b77-82f576701809 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:56.497136 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51d2989-7de6-4255-8f73-ec344ad86edf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:56.497689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51d2989-7de6-4255-8f73-ec344ad86edf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:56.497968 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51d2989-7de6-4255-8f73-ec344ad86edf admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:56.498241 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51d2989-7de6-4255-8f73-ec344ad86edf admin admin] PATH_INFO: `/v3/users/8ce68660cf854c1dacb60b6a7f37f21b` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:56.499365 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51d2989-7de6-4255-8f73-ec344ad86edf admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ce68660cf854c1dacb60b6a7f37f21b)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:56.501319 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51d2989-7de6-4255-8f73-ec344ad86edf admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:56.669854 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f51d2989-7de6-4255-8f73-ec344ad86edf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ce68660cf854c1dacb60b6a7f37f21b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:56.676789 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f51d2989-7de6-4255-8f73-ec344ad86edf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ce68660cf854c1dacb60b6a7f37f21b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:56.677457 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f51d2989-7de6-4255-8f73-ec344ad86edf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ce68660cf854c1dacb60b6a7f37f21b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:56.689610 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f51d2989-7de6-4255-8f73-ec344ad86edf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ce68660cf854c1dacb60b6a7f37f21b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:56.787435 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-f51d2989-7de6-4255-8f73-ec344ad86edf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ce68660cf854c1dacb60b6a7f37f21b'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:56.793906 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1427/2861] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:37:56 2022] DELETE /identity/v3/users/8ce68660cf854c1dacb60b6a7f37f21b => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:37:57.195086 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:57.196700 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:57.210810 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c93f3d-748b-412b-9ffa-2b3c8f546eb4 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:57.274379 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d456cb88-5a6d-4ad4-bd1a-b51fa4bdef8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:57.275008 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d456cb88-5a6d-4ad4-bd1a-b51fa4bdef8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:57.275284 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d456cb88-5a6d-4ad4-bd1a-b51fa4bdef8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:57.275551 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d456cb88-5a6d-4ad4-bd1a-b51fa4bdef8f admin admin] PATH_INFO: `/v3/projects/5559837564614bc4bc5097d31d97a6b3` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:57.276719 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d456cb88-5a6d-4ad4-bd1a-b51fa4bdef8f admin admin] RBAC: Authorizing `identity:delete_project(project_id=5559837564614bc4bc5097d31d97a6b3)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:57.278639 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d456cb88-5a6d-4ad4-bd1a-b51fa4bdef8f admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:57.394808 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d456cb88-5a6d-4ad4-bd1a-b51fa4bdef8f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5559837564614bc4bc5097d31d97a6b3', 'request_id': 'req-d456cb88-5a6d-4ad4-bd1a-b51fa4bdef8f'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:57.473641 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d456cb88-5a6d-4ad4-bd1a-b51fa4bdef8f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5559837564614bc4bc5097d31d97a6b3', 'request_id': 'req-d456cb88-5a6d-4ad4-bd1a-b51fa4bdef8f'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:57.479703 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d456cb88-5a6d-4ad4-bd1a-b51fa4bdef8f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5559837564614bc4bc5097d31d97a6b3', 'request_id': 'req-d456cb88-5a6d-4ad4-bd1a-b51fa4bdef8f'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:57.491571 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d456cb88-5a6d-4ad4-bd1a-b51fa4bdef8f admin admin] The token cache is being invalidate because project 5559837564614bc4bc5097d31d97a6b3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:37:57.491851 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d456cb88-5a6d-4ad4-bd1a-b51fa4bdef8f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:57.492506 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-d456cb88-5a6d-4ad4-bd1a-b51fa4bdef8f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:57.494026 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1435/2862] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:37:57 2022] DELETE /identity/v3/projects/5559837564614bc4bc5097d31d97a6b3 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:57.499988 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc12da56-6e70-4bd1-bc9b-e98a189df9c1 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:57.500503 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc12da56-6e70-4bd1-bc9b-e98a189df9c1 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:57.500626 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc12da56-6e70-4bd1-bc9b-e98a189df9c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:57.520129 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-dc12da56-6e70-4bd1-bc9b-e98a189df9c1 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:57.521332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-dc12da56-6e70-4bd1-bc9b-e98a189df9c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:57.530973 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:DELETE=18,keystone:SELECT=80,keystone:INSERT=2 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:57.582392 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1428/2863] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:57.591349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e5f5f6-11eb-4abb-9513-ae05c2f4cb1b None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:57.591633 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e5f5f6-11eb-4abb-9513-ae05c2f4cb1b None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:57.591906 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e5f5f6-11eb-4abb-9513-ae05c2f4cb1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:57.611786 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:57.618302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-41e5f5f6-11eb-4abb-9513-ae05c2f4cb1b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:57.619578 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-41e5f5f6-11eb-4abb-9513-ae05c2f4cb1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:57.628145 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1436/2864] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:57.632737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc12da56-6e70-4bd1-bc9b-e98a189df9c1 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:57.643126 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc12da56-6e70-4bd1-bc9b-e98a189df9c1 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:57.647940 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a892574-1ba1-4aab-b35b-8c1d3b229759 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:57.648572 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a892574-1ba1-4aab-b35b-8c1d3b229759 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:57.648847 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a892574-1ba1-4aab-b35b-8c1d3b229759 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:57.649117 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a892574-1ba1-4aab-b35b-8c1d3b229759 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:57.649826 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a892574-1ba1-4aab-b35b-8c1d3b229759 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:37:57.650229 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a892574-1ba1-4aab-b35b-8c1d3b229759 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:57.651841 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a892574-1ba1-4aab-b35b-8c1d3b229759 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:57.666825 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1429/2865] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:37:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:37:57.671223 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e5f5f6-11eb-4abb-9513-ae05c2f4cb1b None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:57.682715 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e5f5f6-11eb-4abb-9513-ae05c2f4cb1b None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:57.688017 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8de8b74-0910-40de-97ab-0a2c5d60219d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:57.690573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8de8b74-0910-40de-97ab-0a2c5d60219d admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:57.690573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8de8b74-0910-40de-97ab-0a2c5d60219d admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:57.690573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8de8b74-0910-40de-97ab-0a2c5d60219d admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:57.690573 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8de8b74-0910-40de-97ab-0a2c5d60219d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:57.691821 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8de8b74-0910-40de-97ab-0a2c5d60219d admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:57.806983 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1437/2866] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:37:57 2022] POST /identity/v3/projects => generated 359 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:57.814334 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a892574-1ba1-4aab-b35b-8c1d3b229759 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:57.822582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a892574-1ba1-4aab-b35b-8c1d3b229759 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:57.887921 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e4df56-633a-4fd7-b7d6-96e24f6c7ce7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:57.888476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e4df56-633a-4fd7-b7d6-96e24f6c7ce7 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:57.888801 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e4df56-633a-4fd7-b7d6-96e24f6c7ce7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:57.889023 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e4df56-633a-4fd7-b7d6-96e24f6c7ce7 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:57.889623 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e4df56-633a-4fd7-b7d6-96e24f6c7ce7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:57.891516 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e4df56-633a-4fd7-b7d6-96e24f6c7ce7 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:58.005288 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1430/2867] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:37:57 2022] POST /identity/v3/users => generated 280 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:58.009638 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8de8b74-0910-40de-97ab-0a2c5d60219d admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:58.020355 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8de8b74-0910-40de-97ab-0a2c5d60219d admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:58.024948 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42b6a14-897e-4611-a736-a0ade73234aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:58.025549 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42b6a14-897e-4611-a736-a0ade73234aa admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:58.028177 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42b6a14-897e-4611-a736-a0ade73234aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:58.028177 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42b6a14-897e-4611-a736-a0ade73234aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:58.028177 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42b6a14-897e-4611-a736-a0ade73234aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:58.028478 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42b6a14-897e-4611-a736-a0ade73234aa admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:58.039103 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1438/2868] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:37:58.043364 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e4df56-633a-4fd7-b7d6-96e24f6c7ce7 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:58.053739 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e4df56-633a-4fd7-b7d6-96e24f6c7ce7 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:58.058518 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a73967-202d-4c51-a38e-b0c8e21ff35a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:58.059120 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a73967-202d-4c51-a38e-b0c8e21ff35a admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:58.059388 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a73967-202d-4c51-a38e-b0c8e21ff35a admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:58.059653 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a73967-202d-4c51-a38e-b0c8e21ff35a admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:58.060345 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a73967-202d-4c51-a38e-b0c8e21ff35a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:58.061934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a73967-202d-4c51-a38e-b0c8e21ff35a admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:58.073308 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1431/2869] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:37:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 21 01:37:58.078815 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42b6a14-897e-4611-a736-a0ade73234aa admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:58.092212 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42b6a14-897e-4611-a736-a0ade73234aa admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:58.096880 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e588f7e1-88a9-4ddf-a484-abb148190167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:58.097435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e588f7e1-88a9-4ddf-a484-abb148190167 admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:58.097711 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e588f7e1-88a9-4ddf-a484-abb148190167 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:58.098627 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e588f7e1-88a9-4ddf-a484-abb148190167 admin admin] PATH_INFO: `/v3/projects/b93fe9761f5a40ebb491c10daab1312a/users/90d81624f0e2458d8f9dbc1b4e999eef/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:58.113891 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e588f7e1-88a9-4ddf-a484-abb148190167 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b93fe9761f5a40ebb491c10daab1312a, user_id=90d81624f0e2458d8f9dbc1b4e999eef, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:58.116776 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e588f7e1-88a9-4ddf-a484-abb148190167 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:58.221754 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1439/2870] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:37:58 2022] PUT /identity/v3/projects/b93fe9761f5a40ebb491c10daab1312a/users/90d81624f0e2458d8f9dbc1b4e999eef/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 21 01:37:58.234706 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2da559-e264-4d85-8ec0-df72a6de9731 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:58.234981 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2da559-e264-4d85-8ec0-df72a6de9731 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:58.235302 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2da559-e264-4d85-8ec0-df72a6de9731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:58.263773 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-4b2da559-e264-4d85-8ec0-df72a6de9731 None None] MFA Rules not processed for user `90d81624f0e2458d8f9dbc1b4e999eef`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:58.265028 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-4b2da559-e264-4d85-8ec0-df72a6de9731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:58.331552 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1432/2871] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:58 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:58.369392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e588f7e1-88a9-4ddf-a484-abb148190167 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:58.371089 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-e588f7e1-88a9-4ddf-a484-abb148190167 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:58.428392 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e588f7e1-88a9-4ddf-a484-abb148190167 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:58.432005 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921916fc-8994-4727-8de4-945218d4c2db service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:58.432632 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921916fc-8994-4727-8de4-945218d4c2db service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:58.432906 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921916fc-8994-4727-8de4-945218d4c2db service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:58.433175 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921916fc-8994-4727-8de4-945218d4c2db service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:58.442951 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921916fc-8994-4727-8de4-945218d4c2db service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:58.444422 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921916fc-8994-4727-8de4-945218d4c2db service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:58.453430 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1440/2872] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:58 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:37:58.469972 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2da559-e264-4d85-8ec0-df72a6de9731 None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:58.483872 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:58.537152 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2da559-e264-4d85-8ec0-df72a6de9731 None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:58.541971 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fd025c-e9d7-4b38-95b8-572a207ac319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:58.542605 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fd025c-e9d7-4b38-95b8-572a207ac319 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:58.542888 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fd025c-e9d7-4b38-95b8-572a207ac319 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:58.543161 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fd025c-e9d7-4b38-95b8-572a207ac319 admin admin] PATH_INFO: `/v3/users/90d81624f0e2458d8f9dbc1b4e999eef` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:58.544332 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fd025c-e9d7-4b38-95b8-572a207ac319 admin admin] RBAC: Authorizing `identity:delete_user(user_id=90d81624f0e2458d8f9dbc1b4e999eef)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:58.546395 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fd025c-e9d7-4b38-95b8-572a207ac319 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:58.727291 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-d0fd025c-e9d7-4b38-95b8-572a207ac319 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '90d81624f0e2458d8f9dbc1b4e999eef'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:58.734477 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-d0fd025c-e9d7-4b38-95b8-572a207ac319 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '90d81624f0e2458d8f9dbc1b4e999eef'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:58.735148 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-d0fd025c-e9d7-4b38-95b8-572a207ac319 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '90d81624f0e2458d8f9dbc1b4e999eef'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:58.747417 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-d0fd025c-e9d7-4b38-95b8-572a207ac319 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '90d81624f0e2458d8f9dbc1b4e999eef'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:58.821209 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-d0fd025c-e9d7-4b38-95b8-572a207ac319 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '90d81624f0e2458d8f9dbc1b4e999eef'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:58.827817 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1433/2873] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:37:58 2022] DELETE /identity/v3/users/90d81624f0e2458d8f9dbc1b4e999eef => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:58.832414 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7904e16-bbff-48e1-91da-7be0079bbe6f None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:58.832694 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7904e16-bbff-48e1-91da-7be0079bbe6f None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:58.832962 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7904e16-bbff-48e1-91da-7be0079bbe6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:58.852384 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-d7904e16-bbff-48e1-91da-7be0079bbe6f None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:58.853590 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-d7904e16-bbff-48e1-91da-7be0079bbe6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:58.913913 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1441/2874] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:58.924009 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fd025c-e9d7-4b38-95b8-572a207ac319 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:58.925525 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d0fd025c-e9d7-4b38-95b8-572a207ac319 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:58.985282 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fd025c-e9d7-4b38-95b8-572a207ac319 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:58.989656 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ba1cd5-3208-4440-a4eb-d855862bfe00 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:58.990267 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ba1cd5-3208-4440-a4eb-d855862bfe00 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:58.990544 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ba1cd5-3208-4440-a4eb-d855862bfe00 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:58.990838 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ba1cd5-3208-4440-a4eb-d855862bfe00 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:59.000339 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ba1cd5-3208-4440-a4eb-d855862bfe00 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:59.001769 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ba1cd5-3208-4440-a4eb-d855862bfe00 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:59.011718 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1434/2875] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:37:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:37:59.428705 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7904e16-bbff-48e1-91da-7be0079bbe6f None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:59.430376 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-d7904e16-bbff-48e1-91da-7be0079bbe6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:59.444441 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7904e16-bbff-48e1-91da-7be0079bbe6f None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:59.449255 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573ce6ab-3236-4ee0-ba4c-49ef0e3f9255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:59.449891 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573ce6ab-3236-4ee0-ba4c-49ef0e3f9255 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:59.450173 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573ce6ab-3236-4ee0-ba4c-49ef0e3f9255 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:59.450447 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573ce6ab-3236-4ee0-ba4c-49ef0e3f9255 admin admin] PATH_INFO: `/v3/projects/b93fe9761f5a40ebb491c10daab1312a` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:59.451581 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573ce6ab-3236-4ee0-ba4c-49ef0e3f9255 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b93fe9761f5a40ebb491c10daab1312a)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:59.453520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573ce6ab-3236-4ee0-ba4c-49ef0e3f9255 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:59.553863 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:59.559820 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-573ce6ab-3236-4ee0-ba4c-49ef0e3f9255 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b93fe9761f5a40ebb491c10daab1312a', 'request_id': 'req-573ce6ab-3236-4ee0-ba4c-49ef0e3f9255'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:59.615303 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-573ce6ab-3236-4ee0-ba4c-49ef0e3f9255 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b93fe9761f5a40ebb491c10daab1312a', 'request_id': 'req-573ce6ab-3236-4ee0-ba4c-49ef0e3f9255'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:59.621831 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-573ce6ab-3236-4ee0-ba4c-49ef0e3f9255 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b93fe9761f5a40ebb491c10daab1312a', 'request_id': 'req-573ce6ab-3236-4ee0-ba4c-49ef0e3f9255'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:59.633493 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-573ce6ab-3236-4ee0-ba4c-49ef0e3f9255 admin admin] The token cache is being invalidate because project b93fe9761f5a40ebb491c10daab1312a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:37:59.633773 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-573ce6ab-3236-4ee0-ba4c-49ef0e3f9255 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:59.634515 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-573ce6ab-3236-4ee0-ba4c-49ef0e3f9255 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:37:59.636077 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1442/2876] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:37:59 2022] DELETE /identity/v3/projects/b93fe9761f5a40ebb491c10daab1312a => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:37:59.641034 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8d8e74-bb39-4f2d-9309-7821cbc3543b None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:59.641555 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8d8e74-bb39-4f2d-9309-7821cbc3543b None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:59.641715 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8d8e74-bb39-4f2d-9309-7821cbc3543b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:59.662869 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-4c8d8e74-bb39-4f2d-9309-7821cbc3543b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:59.664150 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-4c8d8e74-bb39-4f2d-9309-7821cbc3543b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:59.693963 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:37:59.725527 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1435/2877] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:59.733063 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6164fa1-5845-465d-84bd-f6f01055a11b None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:59.733342 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6164fa1-5845-465d-84bd-f6f01055a11b None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:59.733614 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6164fa1-5845-465d-84bd-f6f01055a11b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:59.754778 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-f6164fa1-5845-465d-84bd-f6f01055a11b None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:37:59.755996 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-f6164fa1-5845-465d-84bd-f6f01055a11b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:37:59.764341 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1443/2878] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:37:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:37:59.768799 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8d8e74-bb39-4f2d-9309-7821cbc3543b None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:59.779903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8d8e74-bb39-4f2d-9309-7821cbc3543b None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:59.784616 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64901fc-6c43-444a-b500-c41f37c5cd89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:59.785222 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64901fc-6c43-444a-b500-c41f37c5cd89 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:59.785582 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64901fc-6c43-444a-b500-c41f37c5cd89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:59.785856 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64901fc-6c43-444a-b500-c41f37c5cd89 admin admin] PATH_INFO: `/v3/domains` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:59.786496 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64901fc-6c43-444a-b500-c41f37c5cd89 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81805) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 21 01:37:59.786798 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64901fc-6c43-444a-b500-c41f37c5cd89 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:59.788469 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64901fc-6c43-444a-b500-c41f37c5cd89 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:59.800662 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1436/2879] 10.222.0.47 () {58 vars in 1153 bytes} [Sun Aug 21 01:37:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 21 01:37:59.805267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6164fa1-5845-465d-84bd-f6f01055a11b None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:59.818987 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6164fa1-5845-465d-84bd-f6f01055a11b None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:59.823816 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee043558-92c5-4c48-ab95-5ce51b5a02a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:59.824460 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee043558-92c5-4c48-ab95-5ce51b5a02a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:59.824734 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee043558-92c5-4c48-ab95-5ce51b5a02a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:59.824999 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee043558-92c5-4c48-ab95-5ce51b5a02a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:59.825670 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee043558-92c5-4c48-ab95-5ce51b5a02a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:59.827572 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee043558-92c5-4c48-ab95-5ce51b5a02a8 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:37:59.908799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1444/2880] 10.222.0.47 () {60 vars in 1153 bytes} [Sun Aug 21 01:37:59 2022] POST /identity/v3/projects => generated 369 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:37:59.916087 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64901fc-6c43-444a-b500-c41f37c5cd89 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:37:59.926476 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64901fc-6c43-444a-b500-c41f37c5cd89 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:37:59.985267 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027e4c67-a331-42f0-b904-ae4439a90880 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:37:59.985876 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027e4c67-a331-42f0-b904-ae4439a90880 admin admin] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:37:59.986154 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027e4c67-a331-42f0-b904-ae4439a90880 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:37:59.986421 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027e4c67-a331-42f0-b904-ae4439a90880 admin admin] PATH_INFO: `/v3/users` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:37:59.987036 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027e4c67-a331-42f0-b904-ae4439a90880 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:37:59.988978 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027e4c67-a331-42f0-b904-ae4439a90880 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:38:00.069283 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1437/2881] 10.222.0.47 () {60 vars in 1144 bytes} [Sun Aug 21 01:37:59 2022] POST /identity/v3/users => generated 285 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 21 01:38:00.073337 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee043558-92c5-4c48-ab95-5ce51b5a02a8 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:38:00.083859 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee043558-92c5-4c48-ab95-5ce51b5a02a8 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:38:00.088901 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf3f4c8-f597-42f2-b11a-f43af4660950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:38:00.089499 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf3f4c8-f597-42f2-b11a-f43af4660950 admin admin] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:38:00.089769 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf3f4c8-f597-42f2-b11a-f43af4660950 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:38:00.090034 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf3f4c8-f597-42f2-b11a-f43af4660950 admin admin] PATH_INFO: `/v3/roles` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:38:00.090701 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf3f4c8-f597-42f2-b11a-f43af4660950 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:38:00.092316 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf3f4c8-f597-42f2-b11a-f43af4660950 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:38:00.102599 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1445/2882] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:38:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:38:00.106830 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027e4c67-a331-42f0-b904-ae4439a90880 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:38:00.117653 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027e4c67-a331-42f0-b904-ae4439a90880 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:38:00.122226 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc56179f-7f47-45e4-ac84-a34d387595f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:38:00.122819 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc56179f-7f47-45e4-ac84-a34d387595f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:38:00.123090 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc56179f-7f47-45e4-ac84-a34d387595f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:38:00.123481 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc56179f-7f47-45e4-ac84-a34d387595f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:38:00.124138 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc56179f-7f47-45e4-ac84-a34d387595f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:38:00.125741 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc56179f-7f47-45e4-ac84-a34d387595f5 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:38:00.136663 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1438/2883] 10.222.0.47 () {58 vars in 1122 bytes} [Sun Aug 21 01:38:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 21 01:38:00.140968 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf3f4c8-f597-42f2-b11a-f43af4660950 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:38:00.153279 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf3f4c8-f597-42f2-b11a-f43af4660950 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:38:00.158602 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4f92de-49b9-4ec4-98cb-d7478f5f3fdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:38:00.159237 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4f92de-49b9-4ec4-98cb-d7478f5f3fdc admin admin] REQUEST_METHOD: `PUT` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:38:00.160621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4f92de-49b9-4ec4-98cb-d7478f5f3fdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:38:00.160621 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4f92de-49b9-4ec4-98cb-d7478f5f3fdc admin admin] PATH_INFO: `/v3/projects/5d895d8b54ac4d23aaca3e1065b6035f/users/08e7e7eadd5846d29c28e6e3b5c5d76f/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:38:00.177112 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4f92de-49b9-4ec4-98cb-d7478f5f3fdc admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d895d8b54ac4d23aaca3e1065b6035f, user_id=08e7e7eadd5846d29c28e6e3b5c5d76f, role_id=9ee80b012dcd4e5cb7d4b6f02215a0fe)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:38:00.179736 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4f92de-49b9-4ec4-98cb-d7478f5f3fdc admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:38:00.268229 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1446/2884] 10.222.0.47 () {60 vars in 1483 bytes} [Sun Aug 21 01:38:00 2022] PUT /identity/v3/projects/5d895d8b54ac4d23aaca3e1065b6035f/users/08e7e7eadd5846d29c28e6e3b5c5d76f/roles/9ee80b012dcd4e5cb7d4b6f02215a0fe => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:38:00.273523 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcafe67-751f-4d71-9e5c-e1d80a760436 None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:38:00.273903 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcafe67-751f-4d71-9e5c-e1d80a760436 None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:38:00.274291 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcafe67-751f-4d71-9e5c-e1d80a760436 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:38:00.294509 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-2fcafe67-751f-4d71-9e5c-e1d80a760436 None None] MFA Rules not processed for user `b3234aad965e427281c9352b840a0824`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:38:00.295734 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-2fcafe67-751f-4d71-9e5c-e1d80a760436 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:38:00.356934 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1439/2885] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:38:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:38:00.367067 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4f92de-49b9-4ec4-98cb-d7478f5f3fdc admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:38:00.368822 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-2e4f92de-49b9-4ec4-98cb-d7478f5f3fdc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:38:00.427682 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4f92de-49b9-4ec4-98cb-d7478f5f3fdc admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:38:00.431435 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd208fc-390c-4205-a8dc-ef62a80b7930 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:38:00.432043 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd208fc-390c-4205-a8dc-ef62a80b7930 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:38:00.432359 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd208fc-390c-4205-a8dc-ef62a80b7930 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:38:00.432722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd208fc-390c-4205-a8dc-ef62a80b7930 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:38:00.442110 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd208fc-390c-4205-a8dc-ef62a80b7930 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:38:00.443530 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd208fc-390c-4205-a8dc-ef62a80b7930 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:38:00.452975 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1447/2886] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:38:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:38:01.680124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a35eb6-2fa8-48a0-918a-e99d556fa01c None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:38:01.680791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a35eb6-2fa8-48a0-918a-e99d556fa01c None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:38:01.680791 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a35eb6-2fa8-48a0-918a-e99d556fa01c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:38:01.712870 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-f9a35eb6-2fa8-48a0-918a-e99d556fa01c None None] MFA Rules not processed for user `08e7e7eadd5846d29c28e6e3b5c5d76f`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:38:01.714175 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-f9a35eb6-2fa8-48a0-918a-e99d556fa01c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:38:01.746270 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:38:01.783793 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1440/2887] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:38:01 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:38:01.794916 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd208fc-390c-4205-a8dc-ef62a80b7930 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:38:01.796505 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-fdd208fc-390c-4205-a8dc-ef62a80b7930 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:38:01.857450 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd208fc-390c-4205-a8dc-ef62a80b7930 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:38:01.861165 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c20c91-fa19-4b35-b811-d04543b06d00 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b953036b8b774f25bda59cec96b21c27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:38:01.861766 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c20c91-fa19-4b35-b811-d04543b06d00 service cinder] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:38:01.862055 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c20c91-fa19-4b35-b811-d04543b06d00 service cinder] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:38:01.862313 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c20c91-fa19-4b35-b811-d04543b06d00 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:38:01.871398 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c20c91-fa19-4b35-b811-d04543b06d00 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:38:01.872852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c20c91-fa19-4b35-b811-d04543b06d00 service cinder] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:38:01.882918 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1448/2888] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:38:01 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:38:03.598044 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f76e31-86ee-43aa-a0bf-31b51a7130ea None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:38:03.598550 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f76e31-86ee-43aa-a0bf-31b51a7130ea None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:38:03.598682 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f76e31-86ee-43aa-a0bf-31b51a7130ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:38:03.618951 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-d5f76e31-86ee-43aa-a0bf-31b51a7130ea None None] MFA Rules not processed for user `08e7e7eadd5846d29c28e6e3b5c5d76f`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:38:03.620192 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d5f76e31-86ee-43aa-a0bf-31b51a7130ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:38:03.627888 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1441/2889] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:38:03 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:38:11.877985 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=88,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:38:13.613330 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=17 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:38:23.291929 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a82d8e-501c-4ecf-a705-f2440f3de546 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:38:23.291929 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a82d8e-501c-4ecf-a705-f2440f3de546 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:38:23.291929 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a82d8e-501c-4ecf-a705-f2440f3de546 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:38:23.300272 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-c9a82d8e-501c-4ecf-a705-f2440f3de546 None None] MFA Rules not processed for user `08e7e7eadd5846d29c28e6e3b5c5d76f`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:38:23.301452 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-c9a82d8e-501c-4ecf-a705-f2440f3de546 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:38:23.308989 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1449/2890] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:38:23 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:38:33.294819 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=3 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:38:39.687287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f76e31-86ee-43aa-a0bf-31b51a7130ea None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:38:39.688722 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d5f76e31-86ee-43aa-a0bf-31b51a7130ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:38:39.758759 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f76e31-86ee-43aa-a0bf-31b51a7130ea None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:38:39.763889 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a13613-52e4-4add-8c23-a0f25bbea243 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '972b0f1dbfef4d85a6dfc4c5fd839bde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:38:39.764528 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a13613-52e4-4add-8c23-a0f25bbea243 service placement] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:38:39.764803 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a13613-52e4-4add-8c23-a0f25bbea243 service placement] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:38:39.765069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a13613-52e4-4add-8c23-a0f25bbea243 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:38:39.766951 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e8a13613-52e4-4add-8c23-a0f25bbea243 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:38:39.779399 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a13613-52e4-4add-8c23-a0f25bbea243 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:38:39.780994 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a13613-52e4-4add-8c23-a0f25bbea243 service placement] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:38:39.791975 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1442/2891] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:38:39 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:38:46.975433 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f44593-6d1a-405f-a690-76b3eb5e64b1 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:38:46.976009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f44593-6d1a-405f-a690-76b3eb5e64b1 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:38:46.976009 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f44593-6d1a-405f-a690-76b3eb5e64b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:38:47.029841 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-53f44593-6d1a-405f-a690-76b3eb5e64b1 None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:38:47.031257 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-53f44593-6d1a-405f-a690-76b3eb5e64b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:38:47.100714 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1450/2892] 10.222.0.47 () {58 vars in 1009 bytes} [Sun Aug 21 01:38:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:38:47.258680 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a13613-52e4-4add-8c23-a0f25bbea243 service placement] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:38:47.260227 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-e8a13613-52e4-4add-8c23-a0f25bbea243 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:38:47.353964 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a13613-52e4-4add-8c23-a0f25bbea243 service placement] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:38:47.358996 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e003e49d-52d0-441c-9048-d2b5261cccb8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:38:47.359599 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e003e49d-52d0-441c-9048-d2b5261cccb8 service swift] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:38:47.359871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e003e49d-52d0-441c-9048-d2b5261cccb8 service swift] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:38:47.360167 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e003e49d-52d0-441c-9048-d2b5261cccb8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:38:47.368689 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e003e49d-52d0-441c-9048-d2b5261cccb8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:38:47.370277 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e003e49d-52d0-441c-9048-d2b5261cccb8 service swift] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:38:47.380289 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1443/2893] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:38:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 21 01:38:57.097658 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=40 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:38:57.375515 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=73 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:39:04.548349 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0b48c1-8aa2-4b7c-82b2-71597f798d66 None None] REQUEST_METHOD: `POST` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:39:04.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0b48c1-8aa2-4b7c-82b2-71597f798d66 None None] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:39:04.548945 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0b48c1-8aa2-4b7c-82b2-71597f798d66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:39:04.567267 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.auth.core [None req-0b0b48c1-8aa2-4b7c-82b2-71597f798d66 None None] MFA Rules not processed for user `d7c65195ff8d42958fa54b5f083bf8da`. Rule list: `[]` (Enabled: `True`). {{(pid=81804) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:39:04.568477 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-0b0b48c1-8aa2-4b7c-82b2-71597f798d66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:39:04.576023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1451/2894] 10.222.0.47 () {58 vars in 967 bytes} [Sun Aug 21 01:39:04 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:39:04.589815 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e003e49d-52d0-441c-9048-d2b5261cccb8 service swift] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:39:04.600377 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e003e49d-52d0-441c-9048-d2b5261cccb8 service swift] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:39:04.605292 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9e35e7-a6c6-4fb3-aa6d-dc039cfbda63 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06a5972a55824711b06670b925322739', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:39:04.605876 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9e35e7-a6c6-4fb3-aa6d-dc039cfbda63 service swift] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:39:04.606149 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9e35e7-a6c6-4fb3-aa6d-dc039cfbda63 service swift] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:39:04.606415 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9e35e7-a6c6-4fb3-aa6d-dc039cfbda63 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:39:04.614137 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9e35e7-a6c6-4fb3-aa6d-dc039cfbda63 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:39:04.615685 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9e35e7-a6c6-4fb3-aa6d-dc039cfbda63 service swift] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:39:04.623530 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1444/2895] 10.222.0.47 () {60 vars in 1384 bytes} [Sun Aug 21 01:39:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 21 01:39:07.205016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0b48c1-8aa2-4b7c-82b2-71597f798d66 None None] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:39:07.206624 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-0b0b48c1-8aa2-4b7c-82b2-71597f798d66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:39:07.222780 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0b48c1-8aa2-4b7c-82b2-71597f798d66 None None] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:39:07.228038 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820dc491-5670-482e-b17d-e136ded7103e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '972b0f1dbfef4d85a6dfc4c5fd839bde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:39:07.229331 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820dc491-5670-482e-b17d-e136ded7103e service placement] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:39:07.229663 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820dc491-5670-482e-b17d-e136ded7103e service placement] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:39:07.230018 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820dc491-5670-482e-b17d-e136ded7103e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:39:07.231814 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-820dc491-5670-482e-b17d-e136ded7103e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:39:07.243837 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820dc491-5670-482e-b17d-e136ded7103e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:39:07.245525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820dc491-5670-482e-b17d-e136ded7103e service placement] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:39:07.256659 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1452/2896] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:39:07 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:39:07.330672 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9e35e7-a6c6-4fb3-aa6d-dc039cfbda63 service swift] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:39:07.345771 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9e35e7-a6c6-4fb3-aa6d-dc039cfbda63 service swift] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:39:07.350491 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42559b74-b93d-42bb-b186-20bbc7b7c3a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:39:07.355141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42559b74-b93d-42bb-b186-20bbc7b7c3a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:39:07.355141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42559b74-b93d-42bb-b186-20bbc7b7c3a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:39:07.355141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42559b74-b93d-42bb-b186-20bbc7b7c3a7 admin admin] PATH_INFO: `/v3/users/08e7e7eadd5846d29c28e6e3b5c5d76f` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:39:07.355141 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42559b74-b93d-42bb-b186-20bbc7b7c3a7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=08e7e7eadd5846d29c28e6e3b5c5d76f)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:39:07.356779 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42559b74-b93d-42bb-b186-20bbc7b7c3a7 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:39:07.572179 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-42559b74-b93d-42bb-b186-20bbc7b7c3a7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '08e7e7eadd5846d29c28e6e3b5c5d76f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:39:07.582629 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-42559b74-b93d-42bb-b186-20bbc7b7c3a7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '08e7e7eadd5846d29c28e6e3b5c5d76f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:39:07.583392 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-42559b74-b93d-42bb-b186-20bbc7b7c3a7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '08e7e7eadd5846d29c28e6e3b5c5d76f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:39:07.598737 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-42559b74-b93d-42bb-b186-20bbc7b7c3a7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '08e7e7eadd5846d29c28e6e3b5c5d76f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:39:07.676774 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-42559b74-b93d-42bb-b186-20bbc7b7c3a7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '08e7e7eadd5846d29c28e6e3b5c5d76f'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:39:07.683624 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1445/2897] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:39:07 2022] DELETE /identity/v3/users/08e7e7eadd5846d29c28e6e3b5c5d76f => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:39:07.912162 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820dc491-5670-482e-b17d-e136ded7103e service placement] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:39:07.923043 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820dc491-5670-482e-b17d-e136ded7103e service placement] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:39:07.980902 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35798d6e-148b-44a5-bc21-68581c2562fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:39:07.981528 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35798d6e-148b-44a5-bc21-68581c2562fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:39:07.981805 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35798d6e-148b-44a5-bc21-68581c2562fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:39:07.982073 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35798d6e-148b-44a5-bc21-68581c2562fc admin admin] PATH_INFO: `/v3/projects/5d895d8b54ac4d23aaca3e1065b6035f` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:39:07.983180 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35798d6e-148b-44a5-bc21-68581c2562fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d895d8b54ac4d23aaca3e1065b6035f)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:39:07.985107 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35798d6e-148b-44a5-bc21-68581c2562fc admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:39:08.094147 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-35798d6e-148b-44a5-bc21-68581c2562fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d895d8b54ac4d23aaca3e1065b6035f', 'request_id': 'req-35798d6e-148b-44a5-bc21-68581c2562fc'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:39:08.156266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-35798d6e-148b-44a5-bc21-68581c2562fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d895d8b54ac4d23aaca3e1065b6035f', 'request_id': 'req-35798d6e-148b-44a5-bc21-68581c2562fc'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:39:08.162421 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-35798d6e-148b-44a5-bc21-68581c2562fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d895d8b54ac4d23aaca3e1065b6035f', 'request_id': 'req-35798d6e-148b-44a5-bc21-68581c2562fc'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:39:08.174440 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-35798d6e-148b-44a5-bc21-68581c2562fc admin admin] The token cache is being invalidate because project 5d895d8b54ac4d23aaca3e1065b6035f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81804) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:39:08.174722 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-35798d6e-148b-44a5-bc21-68581c2562fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:39:08.175447 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-35798d6e-148b-44a5-bc21-68581c2562fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:39:08.176959 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1453/2898] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:39:07 2022] DELETE /identity/v3/projects/5d895d8b54ac4d23aaca3e1065b6035f => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:39:17.679466 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=35,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:39:18.171317 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=60,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:39:19.865588 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53663712-f881-4d6e-9ddb-3550b2a8e4db None None] REQUEST_METHOD: `POST` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:39:19.866124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53663712-f881-4d6e-9ddb-3550b2a8e4db None None] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:39:19.866124 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53663712-f881-4d6e-9ddb-3550b2a8e4db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:39:19.886287 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.auth.core [None req-53663712-f881-4d6e-9ddb-3550b2a8e4db None None] MFA Rules not processed for user `fef6461d7ca14cab829412ce3b53852a`. Rule list: `[]` (Enabled: `True`). {{(pid=81805) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 21 01:39:19.891421 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-53663712-f881-4d6e-9ddb-3550b2a8e4db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:39:19.941300 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1446/2899] 10.222.0.47 () {58 vars in 958 bytes} [Sun Aug 21 01:39:19 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 21 01:39:20.258419 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35798d6e-148b-44a5-bc21-68581c2562fc admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:39:20.263305 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-35798d6e-148b-44a5-bc21-68581c2562fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:39:20.316874 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35798d6e-148b-44a5-bc21-68581c2562fc admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:39:20.320691 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196bb84f-09af-4139-8bfa-9bbcda9900c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:39:20.321322 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196bb84f-09af-4139-8bfa-9bbcda9900c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:39:20.321582 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196bb84f-09af-4139-8bfa-9bbcda9900c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:39:20.321857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196bb84f-09af-4139-8bfa-9bbcda9900c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:39:20.324281 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-196bb84f-09af-4139-8bfa-9bbcda9900c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:39:20.334412 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196bb84f-09af-4139-8bfa-9bbcda9900c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:39:20.335858 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196bb84f-09af-4139-8bfa-9bbcda9900c3 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:39:20.344852 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1454/2900] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:39:20 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:39:29.934475 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=25 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:39:30.340913 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=26 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:39:31.679402 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53663712-f881-4d6e-9ddb-3550b2a8e4db None None] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:39:31.680842 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-53663712-f881-4d6e-9ddb-3550b2a8e4db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:39:31.756806 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53663712-f881-4d6e-9ddb-3550b2a8e4db None None] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:39:31.762810 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25238fc1-43ec-4719-9c53-6578f325003f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:39:31.763405 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25238fc1-43ec-4719-9c53-6578f325003f service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:39:31.763678 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25238fc1-43ec-4719-9c53-6578f325003f service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:39:31.763945 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25238fc1-43ec-4719-9c53-6578f325003f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:39:31.765658 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-25238fc1-43ec-4719-9c53-6578f325003f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:39:31.775669 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25238fc1-43ec-4719-9c53-6578f325003f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:39:31.777248 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25238fc1-43ec-4719-9c53-6578f325003f service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:39:31.785882 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1447/2901] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:39:31 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:39:41.782021 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=34 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:41:21.532569 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196bb84f-09af-4139-8bfa-9bbcda9900c3 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:41:21.546471 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196bb84f-09af-4139-8bfa-9bbcda9900c3 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:41:21.548508 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a4fc2d-51d4-452b-84a3-95c1ee62ae40 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:41:21.549207 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a4fc2d-51d4-452b-84a3-95c1ee62ae40 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:41:21.549748 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a4fc2d-51d4-452b-84a3-95c1ee62ae40 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:41:21.550093 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a4fc2d-51d4-452b-84a3-95c1ee62ae40 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:41:21.551946 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-82a4fc2d-51d4-452b-84a3-95c1ee62ae40 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:41:21.564287 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a4fc2d-51d4-452b-84a3-95c1ee62ae40 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:41:21.567048 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a4fc2d-51d4-452b-84a3-95c1ee62ae40 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:41:21.587836 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1455/2902] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:41:21 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 21 01:41:21.629144 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25238fc1-43ec-4719-9c53-6578f325003f service nova] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:41:21.639327 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25238fc1-43ec-4719-9c53-6578f325003f service nova] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:41:21.643309 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65733629-2993-4646-ace0-8a6363f90258 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:41:21.643936 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65733629-2993-4646-ace0-8a6363f90258 service neutron] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:41:21.644399 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65733629-2993-4646-ace0-8a6363f90258 service neutron] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:41:21.650355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65733629-2993-4646-ace0-8a6363f90258 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:41:21.650355 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-65733629-2993-4646-ace0-8a6363f90258 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:41:21.661018 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65733629-2993-4646-ace0-8a6363f90258 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:41:21.662444 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65733629-2993-4646-ace0-8a6363f90258 service neutron] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:41:21.674396 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1448/2903] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:41:21 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:41:22.862744 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a4fc2d-51d4-452b-84a3-95c1ee62ae40 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:41:22.871386 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a4fc2d-51d4-452b-84a3-95c1ee62ae40 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:41:22.875151 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858c4603-57a2-422f-93b2-b101fc9144c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:41:22.875734 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858c4603-57a2-422f-93b2-b101fc9144c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:41:22.876004 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858c4603-57a2-422f-93b2-b101fc9144c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:41:22.876299 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858c4603-57a2-422f-93b2-b101fc9144c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:41:22.877999 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-858c4603-57a2-422f-93b2-b101fc9144c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:41:22.891386 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858c4603-57a2-422f-93b2-b101fc9144c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:41:22.892860 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858c4603-57a2-422f-93b2-b101fc9144c4 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:41:22.903758 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1456/2904] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:41:22 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:41:24.666758 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65733629-2993-4646-ace0-8a6363f90258 service neutron] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:41:24.678485 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65733629-2993-4646-ace0-8a6363f90258 service neutron] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:41:24.683925 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6256d60-3955-4184-b786-9dbf1dc66062 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fabdcdc25d1d4ef6bc66bd6cadecaaf8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:41:24.687214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6256d60-3955-4184-b786-9dbf1dc66062 service nova] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:41:24.687214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6256d60-3955-4184-b786-9dbf1dc66062 service nova] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:41:24.687214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6256d60-3955-4184-b786-9dbf1dc66062 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:41:24.687214 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d6256d60-3955-4184-b786-9dbf1dc66062 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:41:24.698815 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6256d60-3955-4184-b786-9dbf1dc66062 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:41:24.700395 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6256d60-3955-4184-b786-9dbf1dc66062 service nova] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:41:24.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1449/2905] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:41:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:41:32.898805 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=12 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:41:34.707186 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=12 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:41:37.365101 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858c4603-57a2-422f-93b2-b101fc9144c4 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:41:37.366548 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-858c4603-57a2-422f-93b2-b101fc9144c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:41:37.377537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858c4603-57a2-422f-93b2-b101fc9144c4 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:41:37.381247 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810fbb46-39c0-4d60-af10-06d1d7c1c4d1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b532ddab1f5e467989edaa9e4b74bd72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:41:37.381840 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810fbb46-39c0-4d60-af10-06d1d7c1c4d1 service neutron] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:41:37.382114 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810fbb46-39c0-4d60-af10-06d1d7c1c4d1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:41:37.382388 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810fbb46-39c0-4d60-af10-06d1d7c1c4d1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:41:37.384066 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-810fbb46-39c0-4d60-af10-06d1d7c1c4d1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:41:37.445579 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810fbb46-39c0-4d60-af10-06d1d7c1c4d1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:41:37.446991 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810fbb46-39c0-4d60-af10-06d1d7c1c4d1 service neutron] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:41:37.456895 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1457/2906] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:41:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:41:40.663279 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6256d60-3955-4184-b786-9dbf1dc66062 service nova] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:41:40.664771 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-d6256d60-3955-4184-b786-9dbf1dc66062 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:41:40.738564 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6256d60-3955-4184-b786-9dbf1dc66062 service nova] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:41:40.743657 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f48a5f8-f854-483e-bd37-8ca39ced6e3b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '972b0f1dbfef4d85a6dfc4c5fd839bde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:41:40.744293 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f48a5f8-f854-483e-bd37-8ca39ced6e3b service placement] REQUEST_METHOD: `GET` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:41:40.744567 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f48a5f8-f854-483e-bd37-8ca39ced6e3b service placement] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:41:40.744848 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f48a5f8-f854-483e-bd37-8ca39ced6e3b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:41:40.747307 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-4f48a5f8-f854-483e-bd37-8ca39ced6e3b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:41:40.759765 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f48a5f8-f854-483e-bd37-8ca39ced6e3b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:41:40.761361 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f48a5f8-f854-483e-bd37-8ca39ced6e3b service placement] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:41:40.771851 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1450/2907] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:41:40 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:41:44.198525 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810fbb46-39c0-4d60-af10-06d1d7c1c4d1 service neutron] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:41:44.200399 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-810fbb46-39c0-4d60-af10-06d1d7c1c4d1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:41:44.212721 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810fbb46-39c0-4d60-af10-06d1d7c1c4d1 service neutron] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:41:44.217314 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe4f1c3-8ad7-4f7f-8f18-eb38d5776c42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:41:44.217986 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe4f1c3-8ad7-4f7f-8f18-eb38d5776c42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:41:44.218302 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe4f1c3-8ad7-4f7f-8f18-eb38d5776c42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:41:44.218537 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe4f1c3-8ad7-4f7f-8f18-eb38d5776c42 admin admin] PATH_INFO: `/v3/users/fef6461d7ca14cab829412ce3b53852a` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:41:44.219660 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe4f1c3-8ad7-4f7f-8f18-eb38d5776c42 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fef6461d7ca14cab829412ce3b53852a)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:41:44.221604 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe4f1c3-8ad7-4f7f-8f18-eb38d5776c42 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:41:44.457406 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9fe4f1c3-8ad7-4f7f-8f18-eb38d5776c42 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fef6461d7ca14cab829412ce3b53852a'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:44.464827 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9fe4f1c3-8ad7-4f7f-8f18-eb38d5776c42 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fef6461d7ca14cab829412ce3b53852a'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:44.479432 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9fe4f1c3-8ad7-4f7f-8f18-eb38d5776c42 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fef6461d7ca14cab829412ce3b53852a'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:44.545071 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9fe4f1c3-8ad7-4f7f-8f18-eb38d5776c42 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fef6461d7ca14cab829412ce3b53852a'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:44.545797 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9fe4f1c3-8ad7-4f7f-8f18-eb38d5776c42 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fef6461d7ca14cab829412ce3b53852a'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:44.553266 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1458/2908] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:41:44 2022] DELETE /identity/v3/users/fef6461d7ca14cab829412ce3b53852a => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:41:44.814069 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f48a5f8-f854-483e-bd37-8ca39ced6e3b service placement] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:41:44.826227 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f48a5f8-f854-483e-bd37-8ca39ced6e3b service placement] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:41:44.885644 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3357a530-9e19-4252-b9d9-8886d8e76d57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:41:44.886303 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3357a530-9e19-4252-b9d9-8886d8e76d57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:41:44.886579 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3357a530-9e19-4252-b9d9-8886d8e76d57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:41:44.886848 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3357a530-9e19-4252-b9d9-8886d8e76d57 admin admin] PATH_INFO: `/v3/projects/4124766f4f2843229059dbbdf7a2de74` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:41:44.887950 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3357a530-9e19-4252-b9d9-8886d8e76d57 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4124766f4f2843229059dbbdf7a2de74)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:41:44.889871 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3357a530-9e19-4252-b9d9-8886d8e76d57 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:41:45.013913 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-3357a530-9e19-4252-b9d9-8886d8e76d57 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4124766f4f2843229059dbbdf7a2de74', 'request_id': 'req-3357a530-9e19-4252-b9d9-8886d8e76d57'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:45.019372 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-3357a530-9e19-4252-b9d9-8886d8e76d57 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4124766f4f2843229059dbbdf7a2de74', 'request_id': 'req-3357a530-9e19-4252-b9d9-8886d8e76d57'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:45.026543 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-3357a530-9e19-4252-b9d9-8886d8e76d57 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4124766f4f2843229059dbbdf7a2de74', 'request_id': 'req-3357a530-9e19-4252-b9d9-8886d8e76d57'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:45.109763 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-3357a530-9e19-4252-b9d9-8886d8e76d57 admin admin] The token cache is being invalidate because project 4124766f4f2843229059dbbdf7a2de74 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:41:45.110050 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-3357a530-9e19-4252-b9d9-8886d8e76d57 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:45.110676 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-3357a530-9e19-4252-b9d9-8886d8e76d57 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:45.112217 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1451/2909] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:41:44 2022] DELETE /identity/v3/projects/4124766f4f2843229059dbbdf7a2de74 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:41:45.116529 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe4f1c3-8ad7-4f7f-8f18-eb38d5776c42 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:41:45.123566 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9fe4f1c3-8ad7-4f7f-8f18-eb38d5776c42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:41:45.130838 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe4f1c3-8ad7-4f7f-8f18-eb38d5776c42 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:41:45.187699 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d36afee-4f90-4af7-82f0-0cd1d774aea3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:41:45.188369 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d36afee-4f90-4af7-82f0-0cd1d774aea3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:41:45.188640 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d36afee-4f90-4af7-82f0-0cd1d774aea3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:41:45.188905 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d36afee-4f90-4af7-82f0-0cd1d774aea3 admin admin] PATH_INFO: `/v3/users/03c22bba97e94049bf33fe365eeceb81` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:41:45.190016 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d36afee-4f90-4af7-82f0-0cd1d774aea3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=03c22bba97e94049bf33fe365eeceb81)` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:41:45.192495 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d36afee-4f90-4af7-82f0-0cd1d774aea3 admin admin] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:41:45.198023 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:41:45.386520 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9d36afee-4f90-4af7-82f0-0cd1d774aea3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '03c22bba97e94049bf33fe365eeceb81'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:45.393377 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9d36afee-4f90-4af7-82f0-0cd1d774aea3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '03c22bba97e94049bf33fe365eeceb81'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:45.405799 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9d36afee-4f90-4af7-82f0-0cd1d774aea3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '03c22bba97e94049bf33fe365eeceb81'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:45.464375 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9d36afee-4f90-4af7-82f0-0cd1d774aea3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '03c22bba97e94049bf33fe365eeceb81'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:45.465044 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.notifications [None req-9d36afee-4f90-4af7-82f0-0cd1d774aea3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '03c22bba97e94049bf33fe365eeceb81'} {{(pid=81804) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:45.471444 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1459/2910] 10.222.0.47 () {58 vars in 1224 bytes} [Sun Aug 21 01:41:45 2022] DELETE /identity/v3/users/03c22bba97e94049bf33fe365eeceb81 => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:41:45.706146 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3357a530-9e19-4252-b9d9-8886d8e76d57 admin admin] Authenticating user token {{(pid=81805) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:41:45.708027 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.fernet_utils [None req-3357a530-9e19-4252-b9d9-8886d8e76d57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81805) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:41:45.721663 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3357a530-9e19-4252-b9d9-8886d8e76d57 admin admin] Validating token access rules against request {{(pid=81805) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:41:45.740804 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:41:45.783915 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893c19b4-54c1-414d-b25d-7578728d3bd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3234aad965e427281c9352b840a0824', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ada79d28652f4a3aa0a174133f24108f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81805) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:41:45.784703 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893c19b4-54c1-414d-b25d-7578728d3bd8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:41:45.785006 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893c19b4-54c1-414d-b25d-7578728d3bd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:41:45.785263 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893c19b4-54c1-414d-b25d-7578728d3bd8 admin admin] PATH_INFO: `/v3/projects/fc202f480d314bad96b99d8bcb31330f` {{(pid=81805) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:41:45.786413 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893c19b4-54c1-414d-b25d-7578728d3bd8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc202f480d314bad96b99d8bcb31330f)` {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:41:45.788368 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893c19b4-54c1-414d-b25d-7578728d3bd8 admin admin] RBAC: Authorization granted {{(pid=81805) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:41:45.889252 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-893c19b4-54c1-414d-b25d-7578728d3bd8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc202f480d314bad96b99d8bcb31330f', 'request_id': 'req-893c19b4-54c1-414d-b25d-7578728d3bd8'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:45.894860 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-893c19b4-54c1-414d-b25d-7578728d3bd8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc202f480d314bad96b99d8bcb31330f', 'request_id': 'req-893c19b4-54c1-414d-b25d-7578728d3bd8'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:45.901230 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-893c19b4-54c1-414d-b25d-7578728d3bd8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc202f480d314bad96b99d8bcb31330f', 'request_id': 'req-893c19b4-54c1-414d-b25d-7578728d3bd8'} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:45.990337 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-893c19b4-54c1-414d-b25d-7578728d3bd8 admin admin] The token cache is being invalidate because project fc202f480d314bad96b99d8bcb31330f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81805) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 21 01:41:45.990759 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-893c19b4-54c1-414d-b25d-7578728d3bd8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:45.991617 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG keystone.notifications [None req-893c19b4-54c1-414d-b25d-7578728d3bd8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81805) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 21 01:41:45.993159 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: [pid: 81805|app: 0|req: 1452/2911] 10.222.0.47 () {58 vars in 1233 bytes} [Sun Aug 21 01:41:45 2022] DELETE /identity/v3/projects/fc202f480d314bad96b99d8bcb31330f => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 21 01:41:55.467768 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=23,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:41:55.985917 dv-cinder-master-3-iscsi devstack@keystone.service[81805]: DEBUG dbcounter [-] [81805] Writing DB stats keystone:SELECT=42,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81805) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 21 01:44:21.463857 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d36afee-4f90-4af7-82f0-0cd1d774aea3 admin admin] Authenticating user token {{(pid=81804) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 21 01:44:21.465346 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-9d36afee-4f90-4af7-82f0-0cd1d774aea3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:44:21.605083 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d36afee-4f90-4af7-82f0-0cd1d774aea3 admin admin] Validating token access rules against request {{(pid=81804) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 21 01:44:21.662556 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f01b26-fb0a-4ce7-a86d-ebef57412e99 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '972b0f1dbfef4d85a6dfc4c5fd839bde', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '75a3bf3f3a4244d18fde47d492657db4', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81804) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 21 01:44:21.663158 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f01b26-fb0a-4ce7-a86d-ebef57412e99 service placement] REQUEST_METHOD: `GET` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 21 01:44:21.663434 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f01b26-fb0a-4ce7-a86d-ebef57412e99 service placement] SCRIPT_NAME: `/identity` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 21 01:44:21.663775 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f01b26-fb0a-4ce7-a86d-ebef57412e99 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81804) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 21 01:44:21.665479 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.fernet_utils [None req-41f01b26-fb0a-4ce7-a86d-ebef57412e99 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81804) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 21 01:44:21.677156 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f01b26-fb0a-4ce7-a86d-ebef57412e99 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 21 01:44:21.678727 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f01b26-fb0a-4ce7-a86d-ebef57412e99 service placement] RBAC: Authorization granted {{(pid=81804) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 21 01:44:21.689442 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: [pid: 81804|app: 0|req: 1460/2912] 10.222.0.47 () {60 vars in 1365 bytes} [Sun Aug 21 01:44:21 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 21 01:44:31.685265 dv-cinder-master-3-iscsi devstack@keystone.service[81804]: DEBUG dbcounter [-] [81804] Writing DB stats keystone:SELECT=89 {{(pid=81804) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}